Nov 01 15:44:59.132135 ip-10-0-143-101 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 15:44:59.132149 ip-10-0-143-101 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 15:44:59.132159 ip-10-0-143-101 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 15:44:59.132502 ip-10-0-143-101 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 15:45:09.235598 ip-10-0-143-101 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 15:45:09.235613 ip-10-0-143-101 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 03a9c7f7ff9e4446abd03f76f79c329c -- Nov 01 15:46:57.257677 ip-10-0-143-101 systemd[1]: Starting Kubernetes Kubelet... Nov 01 15:46:57.655948 ip-10-0-143-101 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:46:57.655948 ip-10-0-143-101 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 15:46:57.655948 ip-10-0-143-101 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:46:57.655948 ip-10-0-143-101 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:46:57.661537 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661434 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.661537 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661532 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.661537 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661537 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.661537 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661542 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.661537 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661547 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661551 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661555 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661559 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661563 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661567 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661570 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661574 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661577 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661581 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661585 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661589 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661593 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661597 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661601 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661604 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661608 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661612 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661616 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661621 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.661852 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661625 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661629 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661634 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661637 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661646 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661650 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661654 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661658 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661662 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661666 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661670 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661674 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661678 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661683 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661688 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661693 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661697 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661702 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661708 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.662697 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661712 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661717 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661721 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661725 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661729 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661734 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661739 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661743 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661747 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661752 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661756 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661760 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661766 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661770 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661774 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661779 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661784 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661788 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661793 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661797 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.663498 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661801 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661805 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661810 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661814 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661819 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661823 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661828 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661833 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661838 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661842 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661847 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661853 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661860 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661865 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661870 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661874 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661878 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661883 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661888 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.664329 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661892 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661898 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661903 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661908 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661912 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661916 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661920 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661925 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661929 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661933 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661937 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661942 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661946 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661950 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661954 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661959 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661963 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661967 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661971 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661976 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.665169 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661981 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661985 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661993 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.661998 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663364 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663374 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663379 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663383 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663404 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663410 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663414 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663419 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663423 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663427 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663433 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663437 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663441 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663445 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663450 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.665750 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663454 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663458 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663463 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663469 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663476 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663483 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663488 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663492 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663496 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663500 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663504 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663508 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663514 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663518 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663522 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663526 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663531 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663535 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663539 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.666216 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663544 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663548 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663552 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663557 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663561 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663565 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663569 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663574 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663579 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663584 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663588 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663592 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663597 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663601 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663606 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663610 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663614 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663619 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663627 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663632 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.666990 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663637 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663641 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663645 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663650 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663654 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663658 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663663 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663667 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663672 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663676 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663680 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663685 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663689 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663693 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663697 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663701 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663706 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663710 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663715 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663719 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.667999 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663724 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663728 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663733 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663738 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663742 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663747 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663751 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663755 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663759 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663764 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663769 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663773 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663777 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663782 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663786 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663790 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663795 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663799 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663803 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663808 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.668848 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663812 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663816 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663821 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663825 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663836 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663841 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663845 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663849 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663853 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663857 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663861 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.663866 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664728 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664737 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664746 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664752 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664757 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664761 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664766 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.669350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664770 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664776 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664781 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664785 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664789 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664794 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664798 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664803 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664807 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664811 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664815 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664820 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664824 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664829 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664833 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664837 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664841 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664845 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664850 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664854 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.670037 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664858 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664862 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664866 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664870 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664874 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664879 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664883 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664888 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664894 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664898 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664902 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664906 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664910 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664914 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664919 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664923 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664927 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664932 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664936 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664940 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.670547 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664944 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664948 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664953 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664958 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664962 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664967 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664971 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664975 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664980 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664984 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664988 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664992 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.664996 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665000 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665004 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665009 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665013 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665017 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665021 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.671047 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665026 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665030 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665035 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665039 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665043 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665047 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665052 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665059 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665063 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665068 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665072 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665076 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665080 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665084 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665088 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665092 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665097 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665102 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665106 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665110 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.671618 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665115 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665119 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665123 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665127 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665132 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665136 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665140 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665149 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665155 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665160 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665164 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665168 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665173 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665176 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665180 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665186 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665190 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665194 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665198 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.672097 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.665206 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666358 2570 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666376 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666402 2570 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666410 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666417 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666422 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666429 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666436 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666441 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666448 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666454 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666459 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666465 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666469 2570 flags.go:64] FLAG: --cgroup-root="" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666474 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666479 2570 flags.go:64] FLAG: --client-ca-file="" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666484 2570 flags.go:64] FLAG: --cloud-provider="external" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666489 2570 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666497 2570 flags.go:64] FLAG: --cluster-domain="" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666502 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666508 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666513 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666518 2570 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 15:46:57.672605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666535 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666540 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666545 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666551 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666555 2570 flags.go:64] FLAG: --contention-profiling="false" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666561 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666566 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666572 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666577 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666584 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666589 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666594 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666599 2570 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666604 2570 flags.go:64] FLAG: --enable-server="true" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666609 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666617 2570 flags.go:64] FLAG: --event-burst="100" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666623 2570 flags.go:64] FLAG: --event-qps="50" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666628 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666633 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666637 2570 flags.go:64] FLAG: --eviction-hard="" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666644 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666649 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666654 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666659 2570 flags.go:64] FLAG: --eviction-soft="" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666664 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 15:46:57.673189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666669 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666674 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666679 2570 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666684 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666689 2570 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666693 2570 flags.go:64] FLAG: --feature-gates="" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666700 2570 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666706 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666711 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666716 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666721 2570 flags.go:64] FLAG: --healthz-port="10248" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666726 2570 flags.go:64] FLAG: --help="false" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666731 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666738 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666743 2570 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666748 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666755 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666761 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666765 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666770 2570 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666775 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666780 2570 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666785 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 15:46:57.673825 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666791 2570 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666795 2570 flags.go:64] FLAG: --kube-reserved="" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666800 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666805 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666810 2570 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666814 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666819 2570 flags.go:64] FLAG: --lock-file="" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666824 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666829 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666834 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666844 2570 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666848 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666853 2570 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666858 2570 flags.go:64] FLAG: --logging-format="text" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666863 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666868 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666874 2570 flags.go:64] FLAG: --manifest-url="" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666878 2570 flags.go:64] FLAG: --manifest-url-header="" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666885 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666890 2570 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666897 2570 flags.go:64] FLAG: --max-pods="110" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666901 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666906 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666912 2570 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666917 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 15:46:57.674379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666923 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666928 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666933 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666945 2570 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666950 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666955 2570 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666961 2570 flags.go:64] FLAG: --pod-cidr="" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666966 2570 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666970 2570 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666975 2570 flags.go:64] FLAG: --pods-per-core="0" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666979 2570 flags.go:64] FLAG: --port="10250" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666984 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666989 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00d842a5f61b98997" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666994 2570 flags.go:64] FLAG: --qos-reserved="" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.666999 2570 flags.go:64] FLAG: --read-only-port="10255" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667003 2570 flags.go:64] FLAG: --register-node="true" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667008 2570 flags.go:64] FLAG: --register-with-taints="" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667014 2570 flags.go:64] FLAG: --registry-burst="10" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667019 2570 flags.go:64] FLAG: --registry-qps="5" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667024 2570 flags.go:64] FLAG: --reserved-cpus="" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667028 2570 flags.go:64] FLAG: --reserved-memory="" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667035 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667040 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667045 2570 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667050 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667054 2570 flags.go:64] FLAG: --runonce="false" Nov 01 15:46:57.675059 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667059 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667064 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667069 2570 flags.go:64] FLAG: --seccomp-default="false" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667074 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667082 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667087 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667093 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667098 2570 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667102 2570 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667107 2570 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667112 2570 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667117 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667122 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667126 2570 flags.go:64] FLAG: --system-cgroups="" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667131 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667139 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667144 2570 flags.go:64] FLAG: --tls-cert-file="" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667149 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667156 2570 flags.go:64] FLAG: --tls-min-version="" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667161 2570 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667165 2570 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667170 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667175 2570 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667180 2570 flags.go:64] FLAG: --v="2" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667187 2570 flags.go:64] FLAG: --version="false" Nov 01 15:46:57.675731 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667194 2570 flags.go:64] FLAG: --vmodule="" Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667200 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667206 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667461 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667469 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667474 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667478 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667483 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667487 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667491 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667496 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667503 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667508 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667512 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667517 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667521 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667526 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667530 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667535 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.676331 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667539 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667544 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667548 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667552 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667557 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667561 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667565 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667569 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667573 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667577 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667582 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667586 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667591 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667595 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667599 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667603 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667609 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667613 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667618 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.676797 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667622 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667627 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667632 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667637 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667641 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667647 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667652 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667656 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667662 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667666 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667671 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667675 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667680 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667684 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667688 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667693 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667697 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667701 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667705 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667710 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.677256 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667714 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667719 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667723 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667729 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667735 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667740 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667745 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667749 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667756 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667763 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667767 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667771 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667776 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667780 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667784 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667788 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667793 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667799 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667804 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.677779 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667808 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667812 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667818 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667823 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667827 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667838 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667842 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667846 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667850 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667854 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667858 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667861 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667865 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667869 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667873 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667877 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667881 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667885 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667889 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667894 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.678249 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667898 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667902 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667907 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667912 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667916 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667920 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667924 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667927 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667931 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667936 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667941 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.667945 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.667996 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668063 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668071 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668076 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.678796 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668081 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668085 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668090 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668094 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668098 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668103 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668107 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668111 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668116 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668120 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668124 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668129 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668134 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668138 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668143 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668147 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668151 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668156 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668163 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.679176 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668169 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668173 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668177 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668182 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668186 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668191 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668195 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668199 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668205 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668209 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668214 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668218 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668223 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668228 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668233 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668237 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668241 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668246 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668251 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.679644 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668256 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668260 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668264 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668269 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668273 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668277 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668281 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668285 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668290 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668295 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668299 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668304 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668310 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668316 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668320 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668325 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668329 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668334 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668339 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668343 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.680100 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668348 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668354 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668358 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668363 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668367 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668372 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668376 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668382 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668409 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668413 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668421 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668428 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668433 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668439 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668443 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668449 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668456 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668461 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668466 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.680599 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668470 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668475 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668480 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668484 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668488 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668495 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668499 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668504 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668508 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668513 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668517 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668521 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668525 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668530 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668536 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668540 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668545 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668549 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668553 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668557 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668562 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.681044 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668567 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668573 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668577 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668581 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.668586 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.677071 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.677089 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.677817 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.677826 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677903 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677911 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677916 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677921 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677926 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677931 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677935 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677939 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677943 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677947 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.681614 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677951 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677956 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677960 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677964 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677968 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677972 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677976 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677980 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677985 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677990 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677993 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.677997 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678001 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678007 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678011 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678015 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678020 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678024 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678027 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.682060 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678034 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678040 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678045 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678050 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678054 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678058 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678062 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678067 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678071 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678075 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678079 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678084 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678088 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678093 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678097 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678101 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678106 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678110 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678115 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678119 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.682526 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678123 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678127 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678132 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678136 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678140 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678144 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678148 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678153 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678157 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678162 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678166 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678170 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678175 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678179 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678183 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678187 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678191 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678195 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678198 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.683027 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678202 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678206 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678210 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678214 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678219 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678223 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678228 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678233 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678237 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678241 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678244 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678248 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678252 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678256 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678261 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678265 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678269 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678274 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678278 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.683485 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678284 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678289 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678295 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678300 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678305 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678309 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678314 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678319 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678323 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678327 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678331 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678336 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678340 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678344 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678348 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678352 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678356 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678360 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.683936 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678363 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.678412 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678630 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678637 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678642 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678646 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678651 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678656 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678660 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678664 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678668 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678672 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678676 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678683 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678688 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.684351 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678693 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678697 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678702 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678707 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678711 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678716 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678721 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678725 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678729 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678733 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678737 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678741 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678745 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678749 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678753 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678757 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678761 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678766 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678770 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678774 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.684723 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678778 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678782 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678786 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678791 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678795 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678799 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678804 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678809 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678812 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678816 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678821 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678825 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678830 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678834 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678838 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678843 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678847 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678851 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678855 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.685197 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678860 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678864 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678868 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678872 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678876 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678881 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678885 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678889 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678893 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678897 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678901 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678905 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678909 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678913 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678918 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678922 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678926 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678930 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678934 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678938 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.685661 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678942 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678946 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678950 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678954 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678959 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678963 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678967 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678971 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678974 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678979 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678983 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678988 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678992 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.678996 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679000 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679004 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679008 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679012 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679016 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679020 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.686130 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679024 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679028 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679032 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679036 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679040 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679044 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679048 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679052 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679056 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679060 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679065 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679071 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679076 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679080 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.679110 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679156 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:46:57.686718 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679162 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679167 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679172 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679176 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679182 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679185 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679191 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679195 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679199 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679203 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679207 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679211 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679215 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679219 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679223 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679227 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679232 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679238 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679243 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679248 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:46:57.687119 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679252 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679257 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679261 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679266 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679270 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679274 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679278 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679282 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679286 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679290 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679295 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679300 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679305 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679309 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679313 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679318 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679322 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679326 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679331 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:46:57.687850 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679340 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679345 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679349 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679353 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679357 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679361 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679365 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679369 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679373 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679377 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679381 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679398 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679404 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679407 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679412 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679416 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679420 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679424 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679428 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679434 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:46:57.688546 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679438 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679441 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679445 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679449 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679454 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679458 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679462 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679466 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679470 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679473 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679477 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679481 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679485 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679490 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679494 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679499 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679502 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679507 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679511 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:46:57.689251 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679515 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679519 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679523 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679528 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679532 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679537 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679541 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679545 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679550 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679554 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679559 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679564 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679568 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679572 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679576 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679583 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679587 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679591 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679595 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679600 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:46:57.690645 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679604 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679608 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679612 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679616 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679620 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679624 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:57.679628 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.679698 2570 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.682455 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.683565 2570 server.go:1014] "Starting client certificate rotation" Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.683665 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.683707 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:46:57.691150 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.685987 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:46:57.692693 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.692671 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:46:57.705865 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.705844 2570 log.go:25] "Validated CRI v1 runtime API" Nov 01 15:46:57.710104 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.710076 2570 log.go:25] "Validated CRI v1 image API" Nov 01 15:46:57.710983 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.710955 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 15:46:57.713285 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.713269 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 15:46:57.714519 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.714500 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8eb0cbf0-b981-49cb-869e-8e9469ca800f:/dev/nvme0n1p4 f6ae0c86-9a61-4e20-b988-6449974d77f3:/dev/nvme0n1p3] Nov 01 15:46:57.714577 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.714520 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 15:46:57.720516 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.720405 2570 manager.go:217] Machine: {Timestamp:2026-11-01 15:46:57.718878369 +0000 UTC m=+0.357249609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100756 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fc9a0f9bec36d5882730fb29f82e SystemUUID:ec26fc9a-0f9b-ec36-d588-2730fb29f82e BootID:03a9c7f7-ff9e-4446-abd0-3f76f79c329c Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:35:ee:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:35:ee:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:98:e7:b0:88:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 15:46:57.720589 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.720518 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 15:46:57.720619 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.720603 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 15:46:57.724016 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.723994 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 15:46:57.724188 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.724016 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-101.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 15:46:57.724851 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.724841 2570 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 15:46:57.724886 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.724853 2570 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 15:46:57.724916 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.724886 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:46:57.724916 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.724899 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:46:57.726183 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.726172 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 15:46:57.730750 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.730739 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 15:46:57.730868 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.730859 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 15:46:57.730916 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.730890 2570 kubelet.go:493] "Attempting to sync node with API server" Nov 01 15:46:57.730916 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.730900 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 15:46:57.730916 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.730913 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 15:46:57.731031 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.730970 2570 kubelet.go:405] "Adding apiserver pod source" Nov 01 15:46:57.731031 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.730980 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 15:46:57.731460 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.731447 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:46:57.734300 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.734280 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 15:46:57.743000 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.742976 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:46:57.744155 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744140 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 15:46:57.744212 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744178 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 15:46:57.744416 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744381 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 15:46:57.744416 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744416 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 15:46:57.744548 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744426 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 15:46:57.744548 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744436 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 15:46:57.744548 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744444 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 15:46:57.744548 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.744452 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 15:46:57.745100 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745091 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 15:46:57.745134 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745103 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 15:46:57.745134 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745113 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 15:46:57.745134 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745120 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 15:46:57.745134 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745134 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 15:46:57.745326 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745146 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 15:46:57.745858 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745848 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 15:46:57.745891 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.745860 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 15:46:57.750419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.750379 2570 server.go:1263] "Started kubelet" Nov 01 15:46:57.750576 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.750549 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 15:46:57.750662 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.750577 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 15:46:57.750712 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.750660 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 15:46:57.751202 ip-10-0-143-101 systemd[1]: Started Kubernetes Kubelet. Nov 01 15:46:57.752086 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.752077 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 15:46:57.754006 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.753990 2570 server.go:317] "Adding debug handlers to kubelet server" Nov 01 15:46:57.754687 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.754669 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-101.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 15:46:57.754687 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.754652 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-101.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 15:46:57.754687 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.754652 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 15:46:57.754897 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.754717 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xf52k" Nov 01 15:46:57.759079 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.759057 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 15:46:57.760727 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.760706 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 15:46:57.761311 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.761297 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 15:46:57.761420 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.761370 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:57.762653 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762637 2570 factory.go:153] Registering CRI-O factory Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762657 2570 factory.go:223] Registration of the crio container factory successfully Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762695 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762714 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762716 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762726 2570 factory.go:55] Registering systemd factory Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762733 2570 factory.go:223] Registration of the systemd container factory successfully Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762747 2570 factory.go:103] Registering Raw factory Nov 01 15:46:57.762758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762764 2570 manager.go:1199] Started watching for new ooms in manager Nov 01 15:46:57.763029 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762783 2570 reconstruct.go:97] "Volume reconstruction finished" Nov 01 15:46:57.763029 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762796 2570 reconciler.go:29] "Reconciler: start to sync state" Nov 01 15:46:57.763029 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762713 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xf52k" Nov 01 15:46:57.763029 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.762887 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 15:46:57.763895 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.763830 2570 manager.go:319] Starting recovery of all containers Nov 01 15:46:57.765228 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.765185 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-101.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 15:46:57.766145 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.765004 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-101.us-east-2.compute.internal.18e3f66b659d8bd9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-101.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-101.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 15:46:57.750346713 +0000 UTC m=+0.388717952,LastTimestamp:2026-11-01 15:46:57.750346713 +0000 UTC m=+0.388717952,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-101.us-east-2.compute.internal,}" Nov 01 15:46:57.775114 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.775091 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:46:57.781172 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.781153 2570 manager.go:324] Recovery completed Nov 01 15:46:57.786353 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.786338 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:46:57.788171 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.788154 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:46:57.788248 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.788185 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:46:57.788248 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.788196 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:46:57.788829 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.788809 2570 cpu_manager.go:226] "Starting" policy="none" Nov 01 15:46:57.788829 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.788822 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 15:46:57.788829 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.788831 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:46:57.788990 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.788839 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 15:46:57.789650 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.789630 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:46:57.789719 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.789662 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:46:57.789719 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.789674 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:46:57.790902 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.790888 2570 policy_none.go:50] "Start" Nov 01 15:46:57.790977 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.790907 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 15:46:57.790977 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.790926 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 15:46:57.793003 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.792989 2570 policy_none.go:44] "Start" Nov 01 15:46:57.822984 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.822960 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 15:46:57.825311 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.825299 2570 manager.go:341] "Starting Device Plugin manager" Nov 01 15:46:57.826347 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.826331 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 15:46:57.826416 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.826348 2570 server.go:87] "Starting device plugin registration server" Nov 01 15:46:57.826709 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.826591 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 15:46:57.826758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.826716 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 15:46:57.826848 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.826823 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 15:46:57.826941 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.826917 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 15:46:57.826941 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.826926 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 15:46:57.827466 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.827445 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 15:46:57.827547 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.827490 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:57.896241 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.896209 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 15:46:57.898268 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.897464 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 15:46:57.898268 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.897482 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 15:46:57.898268 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.897504 2570 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 15:46:57.898268 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.897545 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 15:46:57.901492 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.901467 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:46:57.927591 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.927545 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:46:57.928383 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.928363 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:46:57.928474 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.928407 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:46:57.928474 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.928419 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:46:57.928474 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.928444 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:57.938594 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.938571 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:57.938654 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.938593 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-101.us-east-2.compute.internal\": node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:57.969284 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:57.969264 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:57.997761 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.997736 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal"] Nov 01 15:46:57.997834 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.997814 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:46:57.998645 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.998630 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:46:57.998716 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.998659 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:46:57.998716 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:57.998669 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:46:58.000881 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.000869 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:46:58.001050 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001035 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.001094 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001068 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:46:58.001654 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001627 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:46:58.001654 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001653 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:46:58.001772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001685 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:46:58.001772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001686 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:46:58.001772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001709 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:46:58.001772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.001718 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:46:58.003863 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.003847 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.003961 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.003884 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:46:58.004663 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.004643 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:46:58.004754 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.004673 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:46:58.004754 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.004688 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:46:58.034938 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:58.034921 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" node="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.037716 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:58.037699 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" node="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.064721 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.064700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.064799 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.064725 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.064799 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.064747 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd6f673b54e13544e2f8008a04d167a7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"fd6f673b54e13544e2f8008a04d167a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.069635 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:58.069620 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:58.165896 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.165865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.166037 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.165897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.166037 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.165917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd6f673b54e13544e2f8008a04d167a7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"fd6f673b54e13544e2f8008a04d167a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.166037 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.165951 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd6f673b54e13544e2f8008a04d167a7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"fd6f673b54e13544e2f8008a04d167a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.166037 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.165982 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.166037 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:58.165978 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:58.169982 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:58.169964 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.278464 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.278437 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.345683 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.345640 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:59.348861 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.348839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:46:59.378900 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.378878 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.479505 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.479461 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.580076 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.580008 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.680608 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.680564 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.690736 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.690715 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 15:46:59.690853 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.690836 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 15:46:59.690899 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.690874 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 15:46:59.776104 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.776011 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 15:41:58 +0000 UTC" deadline="2028-05-10 13:21:49.030305323 +0000 UTC" Nov 01 15:46:59.776104 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.776097 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13341h34m49.254212582s" Nov 01 15:46:59.781367 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.781343 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.881956 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.881885 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:46:59.931475 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.931448 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:46:59.932956 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:59.932914 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6f673b54e13544e2f8008a04d167a7.slice/crio-0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d WatchSource:0}: Error finding container 0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d: Status 404 returned error can't find the container with id 0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d Nov 01 15:46:59.937456 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:46:59.937439 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:46:59.942283 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:46:59.942259 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae69c9ba8347dbb2dbf47900d239e09c.slice/crio-241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65 WatchSource:0}: Error finding container 241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65: Status 404 returned error can't find the container with id 241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65 Nov 01 15:46:59.982671 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:46:59.982635 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:47:00.083207 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.083177 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:47:00.183767 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.183695 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:47:00.284549 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.284516 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-101.us-east-2.compute.internal\" not found" Nov 01 15:47:00.348209 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.348185 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:00.370199 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.370168 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:00.382498 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.382475 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:00.383324 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.383306 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:00.388903 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.388889 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:00.739299 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.739269 2570 apiserver.go:52] "Watching apiserver" Nov 01 15:47:00.745809 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.745783 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 15:47:00.746324 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.746293 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal","openshift-image-registry/node-ca-czbft","openshift-multus/multus-additional-cni-plugins-m8qrf","openshift-multus/multus-zn8p6","openshift-network-diagnostics/network-check-target-bdfqx","openshift-dns/node-resolver-wx7cw","kube-system/konnectivity-agent-wvndh","openshift-ovn-kubernetes/ovnkube-node-hvbbz","kube-system/global-pull-secret-syncer-xvq6f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b","openshift-cluster-node-tuning-operator/tuned-v77kh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-l6qq5","openshift-network-operator/iptables-alerter-9x9k2"] Nov 01 15:47:00.748820 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.748797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.752277 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.752250 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:00.752461 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.752443 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 15:47:00.752569 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.752554 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 15:47:00.752705 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.752689 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 15:47:00.752847 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.752831 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5vhjr\"" Nov 01 15:47:00.754357 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.754335 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:00.754457 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.754416 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:00.754683 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.754664 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-28hsn\"" Nov 01 15:47:00.754764 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.754703 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 15:47:00.754764 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.754729 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 15:47:00.757205 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.757188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.759145 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.759125 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:00.759239 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.759194 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:00.760524 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.760505 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 15:47:00.761623 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.761406 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.764161 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.764145 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.764265 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.764210 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:00.766526 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.766508 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.768579 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.768559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.769514 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.769497 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:00.773060 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.771157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.774722 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.774697 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.776697 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.776675 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.779313 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.779290 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 15:47:00.783317 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783290 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vxv\" (UniqueName: \"kubernetes.io/projected/ff05d7d5-9766-49fb-86de-0c02df3dc0af-kube-api-access-49vxv\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.783411 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.783411 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783358 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-modprobe-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.783524 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysconfig\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.783524 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783487 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-var-lib-kubelet\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.783632 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-systemd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.783632 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-node-log\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.783632 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-registration-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.783772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783643 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-dbus\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.783772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783682 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f237b168-098b-4df7-a56e-f245691a8086-host\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.783772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-socket-dir-parent\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.783772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783723 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-host\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.783772 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-config\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783788 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f237b168-098b-4df7-a56e-f245691a8086-serviceca\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783841 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j892r\" (UniqueName: \"kubernetes.io/projected/8b24f7e8-b574-4609-96e5-9908302ac283-kube-api-access-j892r\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783864 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-run\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783914 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-system-cni-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783959 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.783983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-tuned\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.784014 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784007 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-etc-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-script-lib\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mk5k\" (UniqueName: \"kubernetes.io/projected/d72732eb-0380-4f65-ab26-a788f64ad3fd-kube-api-access-7mk5k\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-conf-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-etc-kubernetes\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784142 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt658\" (UniqueName: \"kubernetes.io/projected/7a901c1f-0f50-4d5e-94e8-03326a1c650f-kube-api-access-rt658\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784217 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-conf\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vgf\" (UniqueName: \"kubernetes.io/projected/7039a946-2ec8-414d-9093-7f355172fba4-kube-api-access-x9vgf\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784266 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cnibin\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784287 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-multus\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-env-overrides\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784366 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-netns\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.784428 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784489 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-kubernetes\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-device-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784535 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784564 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zj9w\" (UniqueName: \"kubernetes.io/projected/f237b168-098b-4df7-a56e-f245691a8086-kube-api-access-8zj9w\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784813 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-multus-certs\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784856 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-lib-modules\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-netd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.784964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovn-node-metrics-cert\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785001 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-system-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.785071 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-systemd-units\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-netns\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-var-lib-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785151 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-ovn\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-sys-fs\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-kubelet-config\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785238 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-k8s-cni-cncf-io\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-kubelet\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785334 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-slash\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-bin\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785415 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-socket-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-bin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.785604 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785569 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-systemd\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227hr\" (UniqueName: \"kubernetes.io/projected/2f440e8a-b448-4d18-9306-c5627dd6f5ad-kube-api-access-227hr\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72732eb-0380-4f65-ab26-a788f64ad3fd-tmp-dir\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-konnectivity-ca\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-daemon-config\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785721 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-tmp\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-log-socket\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-agent-certs\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cni-binary-copy\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785872 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-hostroot\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.785979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-sys\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.786004 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-kubelet\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.786029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d72732eb-0380-4f65-ab26-a788f64ad3fd-hosts-file\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.786072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cnibin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.786202 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.786098 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbwmh\" (UniqueName: \"kubernetes.io/projected/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-kube-api-access-gbwmh\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.794227 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.794176 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j2hcw" Nov 01 15:47:00.799769 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.799751 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j2hcw" Nov 01 15:47:00.853767 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.853690 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 15:47:00.853767 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.853701 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 15:47:00.853767 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.853703 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 15:47:00.854003 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.853915 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:00.854003 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.853926 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jsz89\"" Nov 01 15:47:00.854003 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.853943 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjc9m\"" Nov 01 15:47:00.854301 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.854236 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 15:47:00.854301 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.854278 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwtj5\"" Nov 01 15:47:00.854423 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.854409 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 15:47:00.854458 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.854437 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 15:47:00.854458 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.854452 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:00.854536 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.854484 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:00.855279 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855260 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 15:47:00.855373 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855318 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 15:47:00.855373 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855326 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ttxgn\"" Nov 01 15:47:00.855489 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855376 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 15:47:00.855489 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855392 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 15:47:00.855489 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855410 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 15:47:00.855489 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855260 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:00.855489 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855480 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58l9j\"" Nov 01 15:47:00.855689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855575 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 15:47:00.855689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855619 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 15:47:00.855689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855630 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x5r5\"" Nov 01 15:47:00.855689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855661 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 15:47:00.855689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855682 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 15:47:00.855862 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.855711 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 15:47:00.856204 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.856189 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hxd8\"" Nov 01 15:47:00.856297 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.856239 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 15:47:00.874511 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.874487 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 15:47:00.886628 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886759 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886640 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-netd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886759 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovn-node-metrics-cert\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886843 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886751 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-netd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886843 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886797 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886843 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-system-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.886944 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-systemd-units\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886944 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-netns\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886944 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-var-lib-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886944 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-ovn\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.886944 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-sys-fs\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.886944 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886928 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-kubelet-config\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.886944 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-k8s-cni-cncf-io\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886964 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-kubelet\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.886993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-slash\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-bin\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887017 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-ovn\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-socket-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-k8s-cni-cncf-io\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887086 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-system-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887090 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-kubelet\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887119 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-netns\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887122 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-sys-fs\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887143 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-systemd-units\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887154 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-kubelet-config\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-bin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887169 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-var-lib-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887003 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.887196 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-systemd\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227hr\" (UniqueName: \"kubernetes.io/projected/2f440e8a-b448-4d18-9306-c5627dd6f5ad-kube-api-access-227hr\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72732eb-0380-4f65-ab26-a788f64ad3fd-tmp-dir\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-konnectivity-ca\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-daemon-config\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-tmp\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-log-socket\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-host-slash\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887315 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhmx\" (UniqueName: \"kubernetes.io/projected/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-kube-api-access-czhmx\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887347 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-agent-certs\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cni-binary-copy\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-hostroot\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-sys\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887091 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-bin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887494 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-systemd\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.888419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887423 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-kubelet\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887539 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d72732eb-0380-4f65-ab26-a788f64ad3fd-hosts-file\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.887749 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.887858 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs podName:8b24f7e8-b574-4609-96e5-9908302ac283 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:01.387799512 +0000 UTC m=+3.018519777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs") pod "network-metrics-daemon-l6qq5" (UID: "8b24f7e8-b574-4609-96e5-9908302ac283") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.887951 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cnibin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbwmh\" (UniqueName: \"kubernetes.io/projected/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-kube-api-access-gbwmh\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vxv\" (UniqueName: \"kubernetes.io/projected/ff05d7d5-9766-49fb-86de-0c02df3dc0af-kube-api-access-49vxv\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888093 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-sys\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-modprobe-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888217 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888235 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-hostroot\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888239 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-bin\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysconfig\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888275 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-log-socket\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888283 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-slash\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889121 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888298 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-var-lib-kubelet\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-systemd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888344 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-node-log\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888350 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysconfig\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888350 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-kubelet\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888322 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d72732eb-0380-4f65-ab26-a788f64ad3fd-hosts-file\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888390 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-socket-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-registration-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888440 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-modprobe-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-dbus\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f237b168-098b-4df7-a56e-f245691a8086-host\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888512 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-systemd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888522 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cni-binary-copy\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888535 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-daemon-config\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-socket-dir-parent\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-host\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888630 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-registration-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-config\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.889890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888654 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f237b168-098b-4df7-a56e-f245691a8086-host\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888677 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-var-lib-kubelet\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f237b168-098b-4df7-a56e-f245691a8086-serviceca\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888651 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-socket-dir-parent\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888717 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j892r\" (UniqueName: \"kubernetes.io/projected/8b24f7e8-b574-4609-96e5-9908302ac283-kube-api-access-j892r\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888754 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cnibin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888752 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-host\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888791 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888792 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-dbus\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-run\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.888873 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-system-cni-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888903 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-node-log\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.888964 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret podName:e9337a97-e8fa-4061-ba5f-fea449c7d390 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:01.388947673 +0000 UTC m=+3.019667934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret") pod "global-pull-secret-syncer-xvq6f" (UID: "e9337a97-e8fa-4061-ba5f-fea449c7d390") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.888998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889002 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-run\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.890734 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889008 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-system-cni-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-tuned\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889079 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-etc-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-script-lib\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889114 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889131 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mk5k\" (UniqueName: \"kubernetes.io/projected/d72732eb-0380-4f65-ab26-a788f64ad3fd-kube-api-access-7mk5k\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-conf-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-etc-kubernetes\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889199 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt658\" (UniqueName: \"kubernetes.io/projected/7a901c1f-0f50-4d5e-94e8-03326a1c650f-kube-api-access-rt658\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889220 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-conf\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vgf\" (UniqueName: \"kubernetes.io/projected/7039a946-2ec8-414d-9093-7f355172fba4-kube-api-access-x9vgf\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cnibin\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-multus\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-env-overrides\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889348 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-conf-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889362 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-iptables-alerter-script\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.891437 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889388 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-netns\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889429 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cnibin\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889442 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889479 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-etc-kubernetes\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889480 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72732eb-0380-4f65-ab26-a788f64ad3fd-tmp-dir\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889546 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f237b168-098b-4df7-a56e-f245691a8086-serviceca\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889618 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-kubernetes\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889673 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-device-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889725 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zj9w\" (UniqueName: \"kubernetes.io/projected/f237b168-098b-4df7-a56e-f245691a8086-kube-api-access-8zj9w\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-multus-certs\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889779 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889803 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-lib-modules\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.889937 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-lib-modules\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890113 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.892219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890124 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-script-lib\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890144 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-env-overrides\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890163 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-kubernetes\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890204 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890205 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-etc-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890236 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-conf\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890283 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890321 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-netns\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890322 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-multus\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890207 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-device-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890347 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890375 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-multus-certs\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890547 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-config\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890639 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-konnectivity-ca\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890703 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.890832 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.892978 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.891371 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovn-node-metrics-cert\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.893652 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.893629 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-tmp\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.893834 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.893815 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-agent-certs\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:00.896577 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.896557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-tuned\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.898634 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.898610 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227hr\" (UniqueName: \"kubernetes.io/projected/2f440e8a-b448-4d18-9306-c5627dd6f5ad-kube-api-access-227hr\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:00.902113 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.902082 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j892r\" (UniqueName: \"kubernetes.io/projected/8b24f7e8-b574-4609-96e5-9908302ac283-kube-api-access-j892r\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:00.903780 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.903389 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:00.903780 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.903412 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:00.903780 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.903426 2570 projected.go:196] Error preparing data for projected volume kube-api-access-t4vwg for pod openshift-network-diagnostics/network-check-target-bdfqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:00.903780 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:00.903499 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg podName:09cf87aa-e08e-4890-acf5-c8da31cd08da nodeName:}" failed. No retries permitted until 2026-11-01 15:47:01.403475563 +0000 UTC m=+3.034195822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vwg" (UniqueName: "kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg") pod "network-check-target-bdfqx" (UID: "09cf87aa-e08e-4890-acf5-c8da31cd08da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:00.903780 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.903735 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mk5k\" (UniqueName: \"kubernetes.io/projected/d72732eb-0380-4f65-ab26-a788f64ad3fd-kube-api-access-7mk5k\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:00.904103 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.903903 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt658\" (UniqueName: \"kubernetes.io/projected/7a901c1f-0f50-4d5e-94e8-03326a1c650f-kube-api-access-rt658\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:00.906232 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.906208 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbwmh\" (UniqueName: \"kubernetes.io/projected/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-kube-api-access-gbwmh\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:00.908161 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.908137 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vgf\" (UniqueName: \"kubernetes.io/projected/7039a946-2ec8-414d-9093-7f355172fba4-kube-api-access-x9vgf\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:00.910339 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.910304 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zj9w\" (UniqueName: \"kubernetes.io/projected/f237b168-098b-4df7-a56e-f245691a8086-kube-api-access-8zj9w\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:00.911197 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.911177 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vxv\" (UniqueName: \"kubernetes.io/projected/ff05d7d5-9766-49fb-86de-0c02df3dc0af-kube-api-access-49vxv\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:00.911317 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.911269 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"ae69c9ba8347dbb2dbf47900d239e09c","Type":"ContainerStarted","Data":"241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65"} Nov 01 15:47:00.913148 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.913123 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"fd6f673b54e13544e2f8008a04d167a7","Type":"ContainerStarted","Data":"0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d"} Nov 01 15:47:00.991064 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.990965 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-host-slash\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.991064 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.991000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhmx\" (UniqueName: \"kubernetes.io/projected/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-kube-api-access-czhmx\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.991277 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.991086 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-host-slash\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.991277 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.991116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-iptables-alerter-script\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.991611 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.991594 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-iptables-alerter-script\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:00.999569 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:00.999542 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhmx\" (UniqueName: \"kubernetes.io/projected/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-kube-api-access-czhmx\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:01.072461 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.072428 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:01.078758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.078724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:01.083350 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:01.083323 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7039a946_2ec8_414d_9093_7f355172fba4.slice/crio-f56ebf77df15b3f21f331172314cd8afccf9a4ed636db7491ff0591399f71cc8 WatchSource:0}: Error finding container f56ebf77df15b3f21f331172314cd8afccf9a4ed636db7491ff0591399f71cc8: Status 404 returned error can't find the container with id f56ebf77df15b3f21f331172314cd8afccf9a4ed636db7491ff0591399f71cc8 Nov 01 15:47:01.088743 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.088580 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn8p6" Nov 01 15:47:01.088992 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:01.088967 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01f94e8_cf07_45d8_b023_0f87e7d41cdf.slice/crio-bb341de1665cba78f97eccebff37648402746cddbd8e6bce65b57f33ed6ecdfa WatchSource:0}: Error finding container bb341de1665cba78f97eccebff37648402746cddbd8e6bce65b57f33ed6ecdfa: Status 404 returned error can't find the container with id bb341de1665cba78f97eccebff37648402746cddbd8e6bce65b57f33ed6ecdfa Nov 01 15:47:01.094317 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.094284 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:01.098816 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:01.097819 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a901c1f_0f50_4d5e_94e8_03326a1c650f.slice/crio-6433c5acfa2b7f0a4a44db14196a2443c020c4f96ace892d41563527ac62e3f1 WatchSource:0}: Error finding container 6433c5acfa2b7f0a4a44db14196a2443c020c4f96ace892d41563527ac62e3f1: Status 404 returned error can't find the container with id 6433c5acfa2b7f0a4a44db14196a2443c020c4f96ace892d41563527ac62e3f1 Nov 01 15:47:01.105569 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:01.105545 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72732eb_0380_4f65_ab26_a788f64ad3fd.slice/crio-e98087dbe24fa28c1d6fe98985df40ea9e3014555c2642a2e85c2885b79ccde9 WatchSource:0}: Error finding container e98087dbe24fa28c1d6fe98985df40ea9e3014555c2642a2e85c2885b79ccde9: Status 404 returned error can't find the container with id e98087dbe24fa28c1d6fe98985df40ea9e3014555c2642a2e85c2885b79ccde9 Nov 01 15:47:01.117539 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.117515 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:01.119681 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.119658 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:01.122515 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.122495 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:01.134564 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:01.134531 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f440e8a_b448_4d18_9306_c5627dd6f5ad.slice/crio-63cd1ed6713c3d40014fd3e4776931d3bcc693ba960975e885362a3c71b4d4df WatchSource:0}: Error finding container 63cd1ed6713c3d40014fd3e4776931d3bcc693ba960975e885362a3c71b4d4df: Status 404 returned error can't find the container with id 63cd1ed6713c3d40014fd3e4776931d3bcc693ba960975e885362a3c71b4d4df Nov 01 15:47:01.135410 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:01.135388 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3d24bb_ebe0_4623_88d7_77cd90a8b142.slice/crio-6479c4becc1497c24f39a4338481f57c23b75a7cbdf2187c8035fd0128dc6a89 WatchSource:0}: Error finding container 6479c4becc1497c24f39a4338481f57c23b75a7cbdf2187c8035fd0128dc6a89: Status 404 returned error can't find the container with id 6479c4becc1497c24f39a4338481f57c23b75a7cbdf2187c8035fd0128dc6a89 Nov 01 15:47:01.142476 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.142454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:01.144547 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.144528 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:01.154105 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:01.154031 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff05d7d5_9766_49fb_86de_0c02df3dc0af.slice/crio-5718d617383e9595df537f321da8012d69f7641236d0056d0fcc7e84c3ffa8c3 WatchSource:0}: Error finding container 5718d617383e9595df537f321da8012d69f7641236d0056d0fcc7e84c3ffa8c3: Status 404 returned error can't find the container with id 5718d617383e9595df537f321da8012d69f7641236d0056d0fcc7e84c3ffa8c3 Nov 01 15:47:01.174168 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.174138 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:01.249379 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.249305 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:01.393928 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.393890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:01.394107 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.393944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:01.394107 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.394064 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:01.394107 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.394072 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:01.394232 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.394132 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret podName:e9337a97-e8fa-4061-ba5f-fea449c7d390 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:02.394113335 +0000 UTC m=+4.024833587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret") pod "global-pull-secret-syncer-xvq6f" (UID: "e9337a97-e8fa-4061-ba5f-fea449c7d390") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:01.394232 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.394153 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs podName:8b24f7e8-b574-4609-96e5-9908302ac283 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:02.394141751 +0000 UTC m=+4.024862001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs") pod "network-metrics-daemon-l6qq5" (UID: "8b24f7e8-b574-4609-96e5-9908302ac283") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:01.495286 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.495178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:01.495426 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.495335 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:01.495426 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.495354 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:01.495426 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.495364 2570 projected.go:196] Error preparing data for projected volume kube-api-access-t4vwg for pod openshift-network-diagnostics/network-check-target-bdfqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:01.495426 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:01.495422 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg podName:09cf87aa-e08e-4890-acf5-c8da31cd08da nodeName:}" failed. No retries permitted until 2026-11-01 15:47:02.495401253 +0000 UTC m=+4.126121513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vwg" (UniqueName: "kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg") pod "network-check-target-bdfqx" (UID: "09cf87aa-e08e-4890-acf5-c8da31cd08da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:01.803219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.803028 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:00 +0000 UTC" deadline="2028-06-08 16:05:39.375732336 +0000 UTC" Nov 01 15:47:01.803219 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.803166 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14040h18m37.57257131s" Nov 01 15:47:01.918450 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.917358 2570 generic.go:356] "Generic (PLEG): container finished" podID="ae69c9ba8347dbb2dbf47900d239e09c" containerID="4a78469e53da3ad7ade2ed9c8b5883c4ec955caae55cf437bd4678f410beb5c4" exitCode=0 Nov 01 15:47:01.918450 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.917612 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"ae69c9ba8347dbb2dbf47900d239e09c","Type":"ContainerDied","Data":"4a78469e53da3ad7ade2ed9c8b5883c4ec955caae55cf437bd4678f410beb5c4"} Nov 01 15:47:01.922116 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.921999 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x9k2" event={"ID":"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff","Type":"ContainerStarted","Data":"fc84d1fcd7861299dd4b424fa3058fdd8bf0cbd0601ecda51f31e7a81fb84bc9"} Nov 01 15:47:01.925259 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.925200 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"5718d617383e9595df537f321da8012d69f7641236d0056d0fcc7e84c3ffa8c3"} Nov 01 15:47:01.927263 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.927240 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wvndh" event={"ID":"e01f94e8-cf07-45d8-b023-0f87e7d41cdf","Type":"ContainerStarted","Data":"bb341de1665cba78f97eccebff37648402746cddbd8e6bce65b57f33ed6ecdfa"} Nov 01 15:47:01.937763 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.937687 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" event={"ID":"7039a946-2ec8-414d-9093-7f355172fba4","Type":"ContainerStarted","Data":"f56ebf77df15b3f21f331172314cd8afccf9a4ed636db7491ff0591399f71cc8"} Nov 01 15:47:01.943074 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.942593 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"fd6f673b54e13544e2f8008a04d167a7","Type":"ContainerStarted","Data":"598fcd24bfd609286f254f940026addee5e6e751695dc03ef8e5553aff107693"} Nov 01 15:47:01.949971 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.949886 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v77kh" event={"ID":"2f440e8a-b448-4d18-9306-c5627dd6f5ad","Type":"ContainerStarted","Data":"63cd1ed6713c3d40014fd3e4776931d3bcc693ba960975e885362a3c71b4d4df"} Nov 01 15:47:01.966714 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.966609 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerStarted","Data":"6479c4becc1497c24f39a4338481f57c23b75a7cbdf2187c8035fd0128dc6a89"} Nov 01 15:47:01.976483 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.976406 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-czbft" event={"ID":"f237b168-098b-4df7-a56e-f245691a8086","Type":"ContainerStarted","Data":"fd9c772427de48f7c44ad67b56abd3ba59a0c928a6379a2c6772350b6e91c667"} Nov 01 15:47:01.991147 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:01.991031 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wx7cw" event={"ID":"d72732eb-0380-4f65-ab26-a788f64ad3fd","Type":"ContainerStarted","Data":"e98087dbe24fa28c1d6fe98985df40ea9e3014555c2642a2e85c2885b79ccde9"} Nov 01 15:47:02.009902 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.009812 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn8p6" event={"ID":"7a901c1f-0f50-4d5e-94e8-03326a1c650f","Type":"ContainerStarted","Data":"6433c5acfa2b7f0a4a44db14196a2443c020c4f96ace892d41563527ac62e3f1"} Nov 01 15:47:02.403918 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.403159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:02.403918 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.403214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:02.403918 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.403350 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:02.403918 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.403411 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret podName:e9337a97-e8fa-4061-ba5f-fea449c7d390 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:04.403394227 +0000 UTC m=+6.034114495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret") pod "global-pull-secret-syncer-xvq6f" (UID: "e9337a97-e8fa-4061-ba5f-fea449c7d390") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:02.403918 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.403490 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:02.403918 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.403524 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs podName:8b24f7e8-b574-4609-96e5-9908302ac283 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:04.403512857 +0000 UTC m=+6.034233105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs") pod "network-metrics-daemon-l6qq5" (UID: "8b24f7e8-b574-4609-96e5-9908302ac283") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:02.504618 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.504501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:02.505461 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.504807 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:02.505461 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.504830 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:02.505461 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.504843 2570 projected.go:196] Error preparing data for projected volume kube-api-access-t4vwg for pod openshift-network-diagnostics/network-check-target-bdfqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:02.505461 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.504979 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg podName:09cf87aa-e08e-4890-acf5-c8da31cd08da nodeName:}" failed. No retries permitted until 2026-11-01 15:47:04.504918367 +0000 UTC m=+6.135638618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vwg" (UniqueName: "kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg") pod "network-check-target-bdfqx" (UID: "09cf87aa-e08e-4890-acf5-c8da31cd08da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:02.804359 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.804265 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:00 +0000 UTC" deadline="2028-06-22 09:08:07.483997417 +0000 UTC" Nov 01 15:47:02.804359 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.804308 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14369h21m4.679693938s" Nov 01 15:47:02.911633 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.910802 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:02.911633 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.910938 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:02.911633 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.911344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:02.911633 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.911437 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:02.911633 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:02.911504 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:02.911633 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:02.911582 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:03.025720 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:03.025654 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"ae69c9ba8347dbb2dbf47900d239e09c","Type":"ContainerStarted","Data":"d0cd20ea7e1c5a3f4a403907af8d7786d5055fcc8f1ff6895c2ea212b050f5f4"} Nov 01 15:47:03.041336 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:03.041274 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" podStartSLOduration=3.041253813 podStartE2EDuration="3.041253813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:03.040602848 +0000 UTC m=+4.671323111" watchObservedRunningTime="2026-11-01 15:47:03.041253813 +0000 UTC m=+4.671974080" Nov 01 15:47:03.041518 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:03.041481 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" podStartSLOduration=3.041473286 podStartE2EDuration="3.041473286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:01.958313645 +0000 UTC m=+3.589033911" watchObservedRunningTime="2026-11-01 15:47:03.041473286 +0000 UTC m=+4.672193555" Nov 01 15:47:04.032259 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:04.032109 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerStarted","Data":"58ed2727a8111e45c58ba543d2a3a2c56789ba42e698a8726fb837c84cb2f122"} Nov 01 15:47:04.420613 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:04.420535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:04.420613 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:04.420593 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:04.420869 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.420729 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:04.420869 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.420785 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret podName:e9337a97-e8fa-4061-ba5f-fea449c7d390 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:08.420768051 +0000 UTC m=+10.051488310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret") pod "global-pull-secret-syncer-xvq6f" (UID: "e9337a97-e8fa-4061-ba5f-fea449c7d390") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:04.421210 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.421191 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:04.421321 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.421246 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs podName:8b24f7e8-b574-4609-96e5-9908302ac283 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:08.421231681 +0000 UTC m=+10.051951941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs") pod "network-metrics-daemon-l6qq5" (UID: "8b24f7e8-b574-4609-96e5-9908302ac283") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:04.521274 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:04.521238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:04.521507 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.521451 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:04.521507 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.521470 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:04.521507 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.521481 2570 projected.go:196] Error preparing data for projected volume kube-api-access-t4vwg for pod openshift-network-diagnostics/network-check-target-bdfqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:04.521705 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.521535 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg podName:09cf87aa-e08e-4890-acf5-c8da31cd08da nodeName:}" failed. No retries permitted until 2026-11-01 15:47:08.521518103 +0000 UTC m=+10.152238352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vwg" (UniqueName: "kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg") pod "network-check-target-bdfqx" (UID: "09cf87aa-e08e-4890-acf5-c8da31cd08da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:04.908108 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:04.908074 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:04.908270 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.908216 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:04.908630 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:04.908609 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:04.908752 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.908724 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:04.908828 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:04.908792 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:04.908878 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:04.908866 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:05.041433 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:05.040503 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="58ed2727a8111e45c58ba543d2a3a2c56789ba42e698a8726fb837c84cb2f122" exitCode=0 Nov 01 15:47:05.041433 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:05.040598 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerDied","Data":"58ed2727a8111e45c58ba543d2a3a2c56789ba42e698a8726fb837c84cb2f122"} Nov 01 15:47:05.049188 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:05.048647 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wvndh" event={"ID":"e01f94e8-cf07-45d8-b023-0f87e7d41cdf","Type":"ContainerStarted","Data":"d8f395d2778b254df42fa8185e924243fc31c1775124cc0cc02f454301f93465"} Nov 01 15:47:06.194243 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:06.194190 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:06.840791 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:06.840755 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:06.841991 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:06.841964 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:06.856463 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:06.856319 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wvndh" podStartSLOduration=5.182592835 podStartE2EDuration="8.856302035s" totalImagesPullingTime="3.6737092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:05.074071164 +0000 UTC m=+6.704791432" watchObservedRunningTime="2026-11-01 15:47:06.856302035 +0000 UTC m=+8.487022303" Nov 01 15:47:06.907577 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:06.907552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:06.907746 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:06.907724 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:06.908111 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:06.908093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:06.908206 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:06.908189 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:06.908273 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:06.908244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:06.908323 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:06.908297 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:07.059393 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:07.059317 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-czbft" event={"ID":"f237b168-098b-4df7-a56e-f245691a8086","Type":"ContainerStarted","Data":"c6ab7f4a3525b7ef7b111d54ab7fac8c427abf48cce2f54910a92d9463501355"} Nov 01 15:47:07.060268 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:07.060215 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:07.089574 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:07.089521 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-czbft" podStartSLOduration=3.103529412 podStartE2EDuration="8.089503183s" totalImagesPullingTime="4.985973771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:07.074587285 +0000 UTC m=+8.705307553" watchObservedRunningTime="2026-11-01 15:47:07.089503183 +0000 UTC m=+8.720223451" Nov 01 15:47:08.064973 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.064939 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" event={"ID":"7039a946-2ec8-414d-9093-7f355172fba4","Type":"ContainerStarted","Data":"0bc325da1d40eb2c7468d4c5bbb1a9c19388a37d2f6e47ebea653c6dec26bab8"} Nov 01 15:47:08.451992 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.451907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:08.451992 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.451959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:08.452222 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.452138 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:08.452222 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.452200 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret podName:e9337a97-e8fa-4061-ba5f-fea449c7d390 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:16.452182583 +0000 UTC m=+18.082902847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret") pod "global-pull-secret-syncer-xvq6f" (UID: "e9337a97-e8fa-4061-ba5f-fea449c7d390") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:08.452602 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.452589 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:08.452681 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.452639 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs podName:8b24f7e8-b574-4609-96e5-9908302ac283 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:16.452623912 +0000 UTC m=+18.083344158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs") pod "network-metrics-daemon-l6qq5" (UID: "8b24f7e8-b574-4609-96e5-9908302ac283") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:08.552812 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.552773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:08.553026 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.552992 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:08.553124 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.553057 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:08.553124 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.553074 2570 projected.go:196] Error preparing data for projected volume kube-api-access-t4vwg for pod openshift-network-diagnostics/network-check-target-bdfqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:08.553227 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.553142 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg podName:09cf87aa-e08e-4890-acf5-c8da31cd08da nodeName:}" failed. No retries permitted until 2026-11-01 15:47:16.553124439 +0000 UTC m=+18.183844700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vwg" (UniqueName: "kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg") pod "network-check-target-bdfqx" (UID: "09cf87aa-e08e-4890-acf5-c8da31cd08da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:08.902609 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.902557 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 15:47:08.910535 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.910505 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:08.910709 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.910659 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:08.910781 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.910723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:08.910847 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.910822 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:08.911357 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:08.911330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:08.911471 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:08.911452 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:09.070393 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.068998 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wx7cw" event={"ID":"d72732eb-0380-4f65-ab26-a788f64ad3fd","Type":"ContainerStarted","Data":"58c2f1265deaeec24385847061b5225d3b13b49a673df9fb8be8686b07076075"} Nov 01 15:47:09.073417 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.072290 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn8p6" event={"ID":"7a901c1f-0f50-4d5e-94e8-03326a1c650f","Type":"ContainerStarted","Data":"f1ed0ba72297b371a0ebde7c81c22762677af35b304547c6fde492f8ce19ab48"} Nov 01 15:47:09.077033 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.076957 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" event={"ID":"7039a946-2ec8-414d-9093-7f355172fba4","Type":"ContainerStarted","Data":"860bd8cc600fe898760e945107cb2b9021dbc11fdcb4287333cf47ef290909a8"} Nov 01 15:47:09.087158 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.087110 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wx7cw" podStartSLOduration=2.365925324 podStartE2EDuration="10.087095877s" totalImagesPullingTime="7.721170553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:09.086961303 +0000 UTC m=+10.717681570" watchObservedRunningTime="2026-11-01 15:47:09.087095877 +0000 UTC m=+10.717816144" Nov 01 15:47:09.108818 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.108130 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zn8p6" podStartSLOduration=2.462076741 podStartE2EDuration="10.108113575s" totalImagesPullingTime="7.646036834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:09.107762704 +0000 UTC m=+10.738482970" watchObservedRunningTime="2026-11-01 15:47:09.108113575 +0000 UTC m=+10.738833841" Nov 01 15:47:09.840614 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.840261 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T15:47:08.902581572Z","UUID":"c8a4fb71-03ab-4aa7-a198-9ec190bc4583","Handler":null,"Name":"","Endpoint":""} Nov 01 15:47:09.842057 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.842023 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 15:47:09.842189 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:09.842073 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 15:47:10.081646 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:10.081607 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="c4544ee5d4cae955421459b530c9a54f692fb94f6f931e92e7a4639731dc2254" exitCode=0 Nov 01 15:47:10.082122 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:10.081773 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerDied","Data":"c4544ee5d4cae955421459b530c9a54f692fb94f6f931e92e7a4639731dc2254"} Nov 01 15:47:10.905846 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:10.905760 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:10.905846 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:10.905795 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:10.906056 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:10.905760 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:10.906056 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:10.905897 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:10.906056 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:10.905964 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:10.906056 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:10.906006 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:11.086519 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:11.086061 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x9k2" event={"ID":"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff","Type":"ContainerStarted","Data":"2b5d28a3fef8ddd4bd4d1149ce9fa0990596f001004048aabe75a9109e413eb3"} Nov 01 15:47:11.090467 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:11.090403 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" event={"ID":"7039a946-2ec8-414d-9093-7f355172fba4","Type":"ContainerStarted","Data":"b0c9e636f8dcc210d15ef80b2ccd08fe015e50a74eacf5b89a2e690f2c209ed4"} Nov 01 15:47:11.102945 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:11.102890 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9x9k2" podStartSLOduration=4.195017748 podStartE2EDuration="12.102874848s" totalImagesPullingTime="7.9078571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:11.10224295 +0000 UTC m=+12.732963217" watchObservedRunningTime="2026-11-01 15:47:11.102874848 +0000 UTC m=+12.733595116" Nov 01 15:47:11.119094 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:11.119049 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" podStartSLOduration=4.634639342 podStartE2EDuration="13.119025725s" totalImagesPullingTime="8.484386383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:11.118913459 +0000 UTC m=+12.749633726" watchObservedRunningTime="2026-11-01 15:47:11.119025725 +0000 UTC m=+12.749746001" Nov 01 15:47:12.094471 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:12.094431 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v77kh" event={"ID":"2f440e8a-b448-4d18-9306-c5627dd6f5ad","Type":"ContainerStarted","Data":"21ad47ff4efbe240cfad5cc2c3c232251bcdf74d8c6c4381e0692d2a5d0aed5b"} Nov 01 15:47:12.097195 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:12.097161 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="d2af07a4d9e76c3c1c27c9ee0986457a1a81d6dc8828bfd747dff434b619f7d6" exitCode=0 Nov 01 15:47:12.097345 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:12.097249 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerDied","Data":"d2af07a4d9e76c3c1c27c9ee0986457a1a81d6dc8828bfd747dff434b619f7d6"} Nov 01 15:47:12.111530 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:12.111479 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v77kh" podStartSLOduration=2.841106999 podStartE2EDuration="13.111465151s" totalImagesPullingTime="10.270358152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:12.110901196 +0000 UTC m=+13.741621477" watchObservedRunningTime="2026-11-01 15:47:12.111465151 +0000 UTC m=+13.742185417" Nov 01 15:47:12.905579 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:12.905544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:12.905732 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:12.905599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:12.905732 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:12.905610 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:12.905732 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:12.905695 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:12.905880 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:12.905800 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:12.905932 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:12.905903 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:14.105239 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:14.104978 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="64d591b8b791dc0c6f1e4cfed2cf938b4c4853b65a464c927f9053ff21a7dabf" exitCode=0 Nov 01 15:47:14.105239 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:14.105019 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerDied","Data":"64d591b8b791dc0c6f1e4cfed2cf938b4c4853b65a464c927f9053ff21a7dabf"} Nov 01 15:47:14.906376 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:14.906337 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:14.906539 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:14.906482 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:14.906953 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:14.906931 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:14.907084 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:14.907063 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:14.907143 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:14.907135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:14.907248 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:14.907225 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:16.111637 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.111609 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="3b8b7cf252a02e62760dbf5bc9ac9bbc47b0130ff2441a963a12bfa357390605" exitCode=0 Nov 01 15:47:16.112078 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.111694 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerDied","Data":"3b8b7cf252a02e62760dbf5bc9ac9bbc47b0130ff2441a963a12bfa357390605"} Nov 01 15:47:16.114133 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.114100 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"1717f50c173d3ef031790b8d8ecee11323ada48c6a34c4ef8496981f890e4e38"} Nov 01 15:47:16.114230 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.114163 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"944acf446d931b5a0b3044e01b5d63e4da848c123fe6bdffc72e1e29aab0bdb8"} Nov 01 15:47:16.114230 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.114193 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"2c4ce77b277f313d046dee9c688082819bf81548f9f8083e8eed611a33fb8f37"} Nov 01 15:47:16.114230 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.114221 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"bd5044bc89676414dda2b40033dadea7e72e2118c6a8fcf42978086107a8e2b0"} Nov 01 15:47:16.513724 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.513698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:16.513897 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.513735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:16.513897 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.513856 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:16.514011 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.513916 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:16.514011 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.513924 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs podName:8b24f7e8-b574-4609-96e5-9908302ac283 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.513904818 +0000 UTC m=+34.144625064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs") pod "network-metrics-daemon-l6qq5" (UID: "8b24f7e8-b574-4609-96e5-9908302ac283") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:16.514011 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.513986 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret podName:e9337a97-e8fa-4061-ba5f-fea449c7d390 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.513969341 +0000 UTC m=+34.144689595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret") pod "global-pull-secret-syncer-xvq6f" (UID: "e9337a97-e8fa-4061-ba5f-fea449c7d390") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:16.614796 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.614767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:16.614936 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.614882 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:16.614936 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.614897 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:16.614936 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.614906 2570 projected.go:196] Error preparing data for projected volume kube-api-access-t4vwg for pod openshift-network-diagnostics/network-check-target-bdfqx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:16.615067 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.614949 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg podName:09cf87aa-e08e-4890-acf5-c8da31cd08da nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.614937514 +0000 UTC m=+34.245657759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4vwg" (UniqueName: "kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg") pod "network-check-target-bdfqx" (UID: "09cf87aa-e08e-4890-acf5-c8da31cd08da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:16.906365 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.906333 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:16.906506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.906397 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:16.906506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:16.906403 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:16.906592 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.906506 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:16.906648 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.906625 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:16.906709 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:16.906697 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:17.119399 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:17.119310 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="90b75d7debc2809b64bb083b76a9b966a0984bceae31ed4c250cbcb686170d70" exitCode=0 Nov 01 15:47:17.119904 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:17.119402 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerDied","Data":"90b75d7debc2809b64bb083b76a9b966a0984bceae31ed4c250cbcb686170d70"} Nov 01 15:47:17.122464 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:17.122441 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"06cae9f26cc828947c18ec46e37e801f9d79f31825cc2740f58e280fb41512b4"} Nov 01 15:47:17.122535 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:17.122521 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"f011293cd50ef5f0d00fa0c5da3b6ee75175417a07f7f4c6e066b63b4e30e4cb"} Nov 01 15:47:18.128135 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:18.128096 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" event={"ID":"1a3d24bb-ebe0-4623-88d7-77cd90a8b142","Type":"ContainerStarted","Data":"02684fa91c0efa0e6a638fb214b26206582926546c2cd38df040e44eb76548b5"} Nov 01 15:47:18.150648 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:18.150599 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m8qrf" podStartSLOduration=8.176470927 podStartE2EDuration="19.15058782s" totalImagesPullingTime="10.974116893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:18.150312736 +0000 UTC m=+19.781033003" watchObservedRunningTime="2026-11-01 15:47:18.15058782 +0000 UTC m=+19.781308087" Nov 01 15:47:18.906897 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:18.906867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:18.907109 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:18.906979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:18.907109 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:18.906979 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:18.907109 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:18.907066 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:18.907274 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:18.907127 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:18.907274 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:18.907188 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:19.133257 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:19.133223 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"dfc6104bb8b8e81a890f8745e886b0a97f9760726bde8c308d9dae5f4cda1f19"} Nov 01 15:47:20.905617 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:20.905430 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:20.906252 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:20.905468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:20.906252 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:20.905701 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:20.906252 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:20.905807 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:20.906252 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:20.905521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:20.906252 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:20.905966 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:21.140164 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:21.140127 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" event={"ID":"ff05d7d5-9766-49fb-86de-0c02df3dc0af","Type":"ContainerStarted","Data":"f3a7291549af9820035e1459ee08c78dde0a5a28678d508fdc9f8d3d6c5e7df2"} Nov 01 15:47:21.140368 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:21.140347 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:21.140436 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:21.140423 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:21.156248 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:21.156169 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:21.165846 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:21.165811 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" podStartSLOduration=7.505766725 podStartE2EDuration="22.165799924s" totalImagesPullingTime="14.660033199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:21.165449941 +0000 UTC m=+22.796170207" watchObservedRunningTime="2026-11-01 15:47:21.165799924 +0000 UTC m=+22.796520191" Nov 01 15:47:22.021497 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.021471 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xvq6f"] Nov 01 15:47:22.021843 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.021573 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:22.021843 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:22.021662 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:22.025235 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.025208 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6qq5"] Nov 01 15:47:22.025372 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.025273 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdfqx"] Nov 01 15:47:22.025372 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.025338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:22.025473 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:22.025430 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:22.025511 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.025487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:22.025590 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:22.025573 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:22.142591 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.142562 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:22.157929 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:22.157907 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:23.905408 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:23.905375 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:23.905408 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:23.905399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:23.905929 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:23.905502 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:23.905929 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:23.905556 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:23.905929 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:23.905609 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:23.905929 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:23.905682 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:25.905875 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:25.905841 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:25.905875 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:25.905841 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:25.906459 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:25.905841 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:25.906459 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:25.905955 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:25.906459 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:25.906001 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:25.906459 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:25.906070 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:27.906331 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:27.906290 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:27.906739 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:27.906344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:27.906739 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:27.906445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:27.906739 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:27.906445 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6qq5" podUID="8b24f7e8-b574-4609-96e5-9908302ac283" Nov 01 15:47:27.906739 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:27.906553 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvq6f" podUID="e9337a97-e8fa-4061-ba5f-fea449c7d390" Nov 01 15:47:27.906739 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:27.906610 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdfqx" podUID="09cf87aa-e08e-4890-acf5-c8da31cd08da" Nov 01 15:47:28.648381 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.648197 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeReady" Nov 01 15:47:28.648530 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.648457 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 15:47:28.681386 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.681314 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6v9fs"] Nov 01 15:47:28.683275 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.683259 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" Nov 01 15:47:28.700261 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.700232 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5qrs6"] Nov 01 15:47:28.702639 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.702605 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g4sxx"] Nov 01 15:47:28.702742 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.702725 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" Nov 01 15:47:28.704481 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.704466 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j46fh"] Nov 01 15:47:28.704593 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.704575 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" Nov 01 15:47:28.706311 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.706297 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7"] Nov 01 15:47:28.706431 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.706411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:28.710369 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.710349 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv"] Nov 01 15:47:28.710609 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.710592 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.711226 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.711208 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.711387 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.711319 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.711465 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.711386 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 15:47:28.713012 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.712993 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5qhqp"] Nov 01 15:47:28.713382 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.713347 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" Nov 01 15:47:28.716058 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.716027 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7"] Nov 01 15:47:28.716895 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.716235 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.718919 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.718895 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.719463 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.719438 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99"] Nov 01 15:47:28.719565 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.719529 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.722907 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.722886 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fb79fc6b7-gq4fr"] Nov 01 15:47:28.723025 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.723009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.725232 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.725207 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 15:47:28.725367 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.725349 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6498b\"" Nov 01 15:47:28.725764 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.725750 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fz7xr"] Nov 01 15:47:28.725867 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.725851 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.726958 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.726934 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 15:47:28.727506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.727490 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 15:47:28.727843 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.727823 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 15:47:28.727949 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.727931 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kkxnv\"" Nov 01 15:47:28.728404 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.728381 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8ck6h\"" Nov 01 15:47:28.728502 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.728414 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84cf9dbdbb-c9jvz"] Nov 01 15:47:28.728560 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.728518 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.730758 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.730744 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9"] Nov 01 15:47:28.730883 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.730861 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.730981 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.730958 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2665t\"" Nov 01 15:47:28.733221 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.733201 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj"] Nov 01 15:47:28.733607 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.733592 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.735714 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.735696 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69f6694997-vvcm2"] Nov 01 15:47:28.735821 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.735805 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" Nov 01 15:47:28.738505 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.738486 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6v9fs"] Nov 01 15:47:28.738596 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.738540 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.746980 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.746957 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g4sxx"] Nov 01 15:47:28.748133 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.748111 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7"] Nov 01 15:47:28.748225 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.748162 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv"] Nov 01 15:47:28.761541 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.761470 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5qrs6"] Nov 01 15:47:28.762005 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.761893 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99"] Nov 01 15:47:28.768506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.764644 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j46fh"] Nov 01 15:47:28.768506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.764725 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj"] Nov 01 15:47:28.768506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.764752 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9"] Nov 01 15:47:28.768506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.766133 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69f6694997-vvcm2"] Nov 01 15:47:28.768506 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.768260 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cf2mk"] Nov 01 15:47:28.771424 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.771404 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5qhqp"] Nov 01 15:47:28.771571 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.771439 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fz7xr"] Nov 01 15:47:28.771571 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.771546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:28.772076 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.772060 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jx495"] Nov 01 15:47:28.775830 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.775811 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7"] Nov 01 15:47:28.775993 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.775847 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cf2mk"] Nov 01 15:47:28.775993 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.775930 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jx495" Nov 01 15:47:28.779950 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.778512 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jx495"] Nov 01 15:47:28.780011 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.779955 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fb79fc6b7-gq4fr"] Nov 01 15:47:28.781010 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.780990 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84cf9dbdbb-c9jvz"] Nov 01 15:47:28.800409 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800375 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.800521 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800486 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 15:47:28.800677 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800656 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 15:47:28.800749 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800718 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.800852 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800837 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.800908 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800856 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.800908 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800866 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801005 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.800937 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801073 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801027 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801129 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801091 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801129 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:28.801232 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801169 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 15:47:28.801232 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qd88\" (UniqueName: \"kubernetes.io/projected/8a515047-c16a-4352-acaf-16b5bd0dea55-kube-api-access-4qd88\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.801232 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801226 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801380 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801229 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.801380 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801239 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 15:47:28.801380 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801293 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801380 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-default-certificate\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.801380 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801295 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.801380 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801336 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 15:47:28.801380 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801355 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801392 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801405 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8zr\" (UniqueName: \"kubernetes.io/projected/f1f800e1-24b3-4323-97f3-2009efb63d1b-kube-api-access-bw8zr\") pod \"volume-data-source-validator-58c9f5f46d-xs4tv\" (UID: \"f1f800e1-24b3-4323-97f3-2009efb63d1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801407 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801213 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801441 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801444 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801449 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801523 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.801670 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801525 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a515047-c16a-4352-acaf-16b5bd0dea55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.802063 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801885 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.802063 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801940 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:28.802063 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801976 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cea9e9fb-3362-48cf-b2ec-3db89f981464-serving-cert\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.802063 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.801995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.802247 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802079 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvm7b\" (UniqueName: \"kubernetes.io/projected/23a31617-3eb9-499e-a192-f4d24a34e835-kube-api-access-gvm7b\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.802247 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802113 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.802247 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-config\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.802247 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.802398 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802277 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.802398 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.802398 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.802505 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23a31617-3eb9-499e-a192-f4d24a34e835-serving-cert\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.802546 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsgh4\" (UniqueName: \"kubernetes.io/projected/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04-kube-api-access-vsgh4\") pod \"network-check-source-6f78d59f99-9bwcj\" (UID: \"4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" Nov 01 15:47:28.802578 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802540 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vfx8\" (UniqueName: \"kubernetes.io/projected/a9662ce2-6f58-4385-8f20-f9ed018fea61-kube-api-access-8vfx8\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.802610 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqng2\" (UniqueName: \"kubernetes.io/projected/3be78730-9048-43e9-9d31-e3819027d095-kube-api-access-kqng2\") pod \"kas-connection-checker-685895c7c9-5qrs6\" (UID: \"3be78730-9048-43e9-9d31-e3819027d095\") " pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" Nov 01 15:47:28.802663 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-stats-auth\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.802699 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rj4g\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.802745 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.802778 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-tmp\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.802806 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19f38b5c-f14d-48fd-b0cf-a2581c62384a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:28.802836 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802814 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57440ce3-722e-4835-927a-e8f2fac8b3c7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.802870 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.802870 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.802927 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.802927 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.802999 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.802999 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802967 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcswr\" (UniqueName: \"kubernetes.io/projected/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-kube-api-access-gcswr\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.802999 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.802987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.803171 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.803171 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803129 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.803233 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m465\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.803273 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803238 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a515047-c16a-4352-acaf-16b5bd0dea55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.803317 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803269 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.803317 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zldzc\" (UniqueName: \"kubernetes.io/projected/85d4d845-68cc-44e8-9503-5969ed5fcc1b-kube-api-access-zldzc\") pod \"kas-connection-checker-685895c7c9-g4sxx\" (UID: \"85d4d845-68cc-44e8-9503-5969ed5fcc1b\") " pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" Nov 01 15:47:28.803413 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9662ce2-6f58-4385-8f20-f9ed018fea61-serving-cert\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.803413 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803362 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bd5\" (UniqueName: \"kubernetes.io/projected/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-kube-api-access-f6bd5\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.803522 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-snapshots\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.803522 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.803522 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.803681 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.803681 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfn8\" (UniqueName: \"kubernetes.io/projected/cea9e9fb-3362-48cf-b2ec-3db89f981464-kube-api-access-xmfn8\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.803681 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4tt\" (UniqueName: \"kubernetes.io/projected/1a7cd5bd-620b-4d98-b64a-4582788161a4-kube-api-access-rj4tt\") pod \"kas-connection-checker-685895c7c9-6v9fs\" (UID: \"1a7cd5bd-620b-4d98-b64a-4582788161a4\") " pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" Nov 01 15:47:28.803681 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48zp\" (UniqueName: \"kubernetes.io/projected/57440ce3-722e-4835-927a-e8f2fac8b3c7-kube-api-access-p48zp\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.803942 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9662ce2-6f58-4385-8f20-f9ed018fea61-config\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.803942 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.803700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.808289 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.808273 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 15:47:28.816006 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.815984 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 15:47:28.816823 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.816806 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 15:47:28.816949 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.816937 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 15:47:28.816987 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.816962 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 15:47:28.817609 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817590 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 15:47:28.817686 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817628 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 15:47:28.817736 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817710 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 15:47:28.817736 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817719 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 15:47:28.817736 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817728 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 15:47:28.817860 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817773 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 15:47:28.817860 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817775 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-slghr\"" Nov 01 15:47:28.817860 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817748 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8bq96\"" Nov 01 15:47:28.817860 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817715 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 15:47:28.818062 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817885 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 15:47:28.818062 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817930 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 15:47:28.818062 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.817994 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-srrqq\"" Nov 01 15:47:28.818180 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.818103 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2mxnc\"" Nov 01 15:47:28.818314 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.818295 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 15:47:28.818674 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.818655 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wl7pm\"" Nov 01 15:47:28.904094 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcswr\" (UniqueName: \"kubernetes.io/projected/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-kube-api-access-gcswr\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904161 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904185 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m465\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904203 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a515047-c16a-4352-acaf-16b5bd0dea55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.904223 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.904245 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:28.904250 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:28.904636 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.904636 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.904310 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.404290219 +0000 UTC m=+31.035010484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:28.904636 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.904389 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.404373268 +0000 UTC m=+31.035093532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:28.904636 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.904454 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:28.904636 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.904520 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls podName:57440ce3-722e-4835-927a-e8f2fac8b3c7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.40450308 +0000 UTC m=+31.035223326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-59xc9" (UID: "57440ce3-722e-4835-927a-e8f2fac8b3c7") : secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:28.904809 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zldzc\" (UniqueName: \"kubernetes.io/projected/85d4d845-68cc-44e8-9503-5969ed5fcc1b-kube-api-access-zldzc\") pod \"kas-connection-checker-685895c7c9-g4sxx\" (UID: \"85d4d845-68cc-44e8-9503-5969ed5fcc1b\") " pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" Nov 01 15:47:28.904809 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9662ce2-6f58-4385-8f20-f9ed018fea61-serving-cert\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.904809 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904702 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a515047-c16a-4352-acaf-16b5bd0dea55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.904809 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bd5\" (UniqueName: \"kubernetes.io/projected/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-kube-api-access-f6bd5\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.904809 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-snapshots\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.904975 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904925 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.904975 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.905091 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.904985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.905162 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfn8\" (UniqueName: \"kubernetes.io/projected/cea9e9fb-3362-48cf-b2ec-3db89f981464-kube-api-access-xmfn8\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.905403 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905377 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.905487 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905454 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-snapshots\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.905543 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905498 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.905543 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905513 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4tt\" (UniqueName: \"kubernetes.io/projected/1a7cd5bd-620b-4d98-b64a-4582788161a4-kube-api-access-rj4tt\") pod \"kas-connection-checker-685895c7c9-6v9fs\" (UID: \"1a7cd5bd-620b-4d98-b64a-4582788161a4\") " pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" Nov 01 15:47:28.905641 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48zp\" (UniqueName: \"kubernetes.io/projected/57440ce3-722e-4835-927a-e8f2fac8b3c7-kube-api-access-p48zp\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.905641 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e085d33-5390-4d2e-a616-61160ee87e46-tmp-dir\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:28.905759 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9662ce2-6f58-4385-8f20-f9ed018fea61-config\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.905759 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.905759 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:28.905759 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qd88\" (UniqueName: \"kubernetes.io/projected/8a515047-c16a-4352-acaf-16b5bd0dea55-kube-api-access-4qd88\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.905959 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905757 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-default-certificate\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.905959 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8zr\" (UniqueName: \"kubernetes.io/projected/f1f800e1-24b3-4323-97f3-2009efb63d1b-kube-api-access-bw8zr\") pod \"volume-data-source-validator-58c9f5f46d-xs4tv\" (UID: \"f1f800e1-24b3-4323-97f3-2009efb63d1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" Nov 01 15:47:28.905959 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.905959 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a515047-c16a-4352-acaf-16b5bd0dea55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.905959 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.905851 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 15:47:28.905959 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905861 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.905959 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.905951 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.906193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9662ce2-6f58-4385-8f20-f9ed018fea61-config\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.906436 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.906907 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert podName:19f38b5c-f14d-48fd-b0cf-a2581c62384a nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.40688996 +0000 UTC m=+31.037610208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j46fh" (UID: "19f38b5c-f14d-48fd-b0cf-a2581c62384a") : secret "networking-console-plugin-cert" not found Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.906946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cea9e9fb-3362-48cf-b2ec-3db89f981464-serving-cert\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.906974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm7b\" (UniqueName: \"kubernetes.io/projected/23a31617-3eb9-499e-a192-f4d24a34e835-kube-api-access-gvm7b\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907051 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4mn\" (UniqueName: \"kubernetes.io/projected/0e085d33-5390-4d2e-a616-61160ee87e46-kube-api-access-pw4mn\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907112 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-config\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907200 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23a31617-3eb9-499e-a192-f4d24a34e835-serving-cert\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.907587 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907293 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgh4\" (UniqueName: \"kubernetes.io/projected/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04-kube-api-access-vsgh4\") pod \"network-check-source-6f78d59f99-9bwcj\" (UID: \"4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907322 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vfx8\" (UniqueName: \"kubernetes.io/projected/a9662ce2-6f58-4385-8f20-f9ed018fea61-kube-api-access-8vfx8\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907347 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqng2\" (UniqueName: \"kubernetes.io/projected/3be78730-9048-43e9-9d31-e3819027d095-kube-api-access-kqng2\") pod \"kas-connection-checker-685895c7c9-5qrs6\" (UID: \"3be78730-9048-43e9-9d31-e3819027d095\") " pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e085d33-5390-4d2e-a616-61160ee87e46-config-volume\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-stats-auth\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rj4g\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907492 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-tmp\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19f38b5c-f14d-48fd-b0cf-a2581c62384a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57440ce3-722e-4835-927a-e8f2fac8b3c7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907605 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6d982a-e0d9-42ee-b342-290a02120b36-cert\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907654 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598sq\" (UniqueName: \"kubernetes.io/projected/8f6d982a-e0d9-42ee-b342-290a02120b36-kube-api-access-598sq\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.908325 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.907695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.908959 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.907793 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 15:47:28.908959 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.907847 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls podName:f70b67b3-d8eb-4065-8650-b26de9bc9f7c nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.407833422 +0000 UTC m=+31.038553680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5rr7" (UID: "f70b67b3-d8eb-4065-8650-b26de9bc9f7c") : secret "samples-operator-tls" not found Nov 01 15:47:28.909175 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.909153 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:28.909175 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.909162 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.909175 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.909176 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:28.909325 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.909231 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.409216536 +0000 UTC m=+31.039936794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:28.909325 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.909239 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:28.909325 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:28.909279 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.409269176 +0000 UTC m=+31.039989422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:28.909526 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.909354 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9662ce2-6f58-4385-8f20-f9ed018fea61-serving-cert\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.909526 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.909410 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a515047-c16a-4352-acaf-16b5bd0dea55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.909631 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.909539 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-default-certificate\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.910170 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.910023 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.910994 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.910964 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.911893 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.911870 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.911988 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.911871 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.912179 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.912158 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-tmp\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.913594 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.912415 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.913594 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.912739 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19f38b5c-f14d-48fd-b0cf-a2581c62384a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:28.913594 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.913258 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-config\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.913594 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.913507 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57440ce3-722e-4835-927a-e8f2fac8b3c7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.914104 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.913923 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.915388 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.915359 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cea9e9fb-3362-48cf-b2ec-3db89f981464-serving-cert\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.916075 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.916027 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.916479 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.916454 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bd5\" (UniqueName: \"kubernetes.io/projected/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-kube-api-access-f6bd5\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:28.917191 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.916812 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4tt\" (UniqueName: \"kubernetes.io/projected/1a7cd5bd-620b-4d98-b64a-4582788161a4-kube-api-access-rj4tt\") pod \"kas-connection-checker-685895c7c9-6v9fs\" (UID: \"1a7cd5bd-620b-4d98-b64a-4582788161a4\") " pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" Nov 01 15:47:28.918011 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.917908 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zldzc\" (UniqueName: \"kubernetes.io/projected/85d4d845-68cc-44e8-9503-5969ed5fcc1b-kube-api-access-zldzc\") pod \"kas-connection-checker-685895c7c9-g4sxx\" (UID: \"85d4d845-68cc-44e8-9503-5969ed5fcc1b\") " pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" Nov 01 15:47:28.920147 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.918306 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcswr\" (UniqueName: \"kubernetes.io/projected/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-kube-api-access-gcswr\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.920147 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.918463 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.920147 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.919174 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm7b\" (UniqueName: \"kubernetes.io/projected/23a31617-3eb9-499e-a192-f4d24a34e835-kube-api-access-gvm7b\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.920147 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.919717 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.920147 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.919833 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m465\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.920147 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.920014 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfn8\" (UniqueName: \"kubernetes.io/projected/cea9e9fb-3362-48cf-b2ec-3db89f981464-kube-api-access-xmfn8\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:28.921466 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.921028 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23a31617-3eb9-499e-a192-f4d24a34e835-serving-cert\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:28.922349 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.922322 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-stats-auth\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:28.922445 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.922326 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:28.922519 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.922497 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vfx8\" (UniqueName: \"kubernetes.io/projected/a9662ce2-6f58-4385-8f20-f9ed018fea61-kube-api-access-8vfx8\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:28.923858 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.923838 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsgh4\" (UniqueName: \"kubernetes.io/projected/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04-kube-api-access-vsgh4\") pod \"network-check-source-6f78d59f99-9bwcj\" (UID: \"4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" Nov 01 15:47:28.924407 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.924361 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqng2\" (UniqueName: \"kubernetes.io/projected/3be78730-9048-43e9-9d31-e3819027d095-kube-api-access-kqng2\") pod \"kas-connection-checker-685895c7c9-5qrs6\" (UID: \"3be78730-9048-43e9-9d31-e3819027d095\") " pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" Nov 01 15:47:28.924502 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.924461 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.924502 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.924466 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48zp\" (UniqueName: \"kubernetes.io/projected/57440ce3-722e-4835-927a-e8f2fac8b3c7-kube-api-access-p48zp\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:28.924739 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.924719 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qd88\" (UniqueName: \"kubernetes.io/projected/8a515047-c16a-4352-acaf-16b5bd0dea55-kube-api-access-4qd88\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:28.925271 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.925249 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8zr\" (UniqueName: \"kubernetes.io/projected/f1f800e1-24b3-4323-97f3-2009efb63d1b-kube-api-access-bw8zr\") pod \"volume-data-source-validator-58c9f5f46d-xs4tv\" (UID: \"f1f800e1-24b3-4323-97f3-2009efb63d1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" Nov 01 15:47:28.926149 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.926133 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rj4g\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:28.993413 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:28.993381 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" Nov 01 15:47:29.008842 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.008802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6d982a-e0d9-42ee-b342-290a02120b36-cert\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:29.009082 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.008849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598sq\" (UniqueName: \"kubernetes.io/projected/8f6d982a-e0d9-42ee-b342-290a02120b36-kube-api-access-598sq\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:29.009082 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.009067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.009262 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.009143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e085d33-5390-4d2e-a616-61160ee87e46-tmp-dir\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.009262 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.009232 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4mn\" (UniqueName: \"kubernetes.io/projected/0e085d33-5390-4d2e-a616-61160ee87e46-kube-api-access-pw4mn\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.009262 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.009256 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:29.009503 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.009279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e085d33-5390-4d2e-a616-61160ee87e46-config-volume\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.009503 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.009314 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:29.509296493 +0000 UTC m=+31.140016754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:29.010269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.009795 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e085d33-5390-4d2e-a616-61160ee87e46-config-volume\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.010269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.009967 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e085d33-5390-4d2e-a616-61160ee87e46-tmp-dir\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.011927 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.011910 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6d982a-e0d9-42ee-b342-290a02120b36-cert\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:29.014853 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.014834 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" Nov 01 15:47:29.017236 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.017216 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-598sq\" (UniqueName: \"kubernetes.io/projected/8f6d982a-e0d9-42ee-b342-290a02120b36-kube-api-access-598sq\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:29.017533 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.017517 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4mn\" (UniqueName: \"kubernetes.io/projected/0e085d33-5390-4d2e-a616-61160ee87e46-kube-api-access-pw4mn\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.025439 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.025418 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" Nov 01 15:47:29.037991 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.037957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:29.045207 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.045184 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" Nov 01 15:47:29.051610 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.051387 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:29.076254 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.076010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:29.107512 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.106281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:29.123660 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.123538 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" Nov 01 15:47:29.141278 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.141069 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:29.283204 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.283178 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6v9fs"] Nov 01 15:47:29.309949 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.309923 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5qrs6"] Nov 01 15:47:29.318729 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:29.318685 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be78730_9048_43e9_9d31_e3819027d095.slice/crio-29210824966f85f75f01690b769716d5e8887c2b8f87d4f2c4082b717e52b402 WatchSource:0}: Error finding container 29210824966f85f75f01690b769716d5e8887c2b8f87d4f2c4082b717e52b402: Status 404 returned error can't find the container with id 29210824966f85f75f01690b769716d5e8887c2b8f87d4f2c4082b717e52b402 Nov 01 15:47:29.333901 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.333837 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g4sxx"] Nov 01 15:47:29.336833 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.336801 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7"] Nov 01 15:47:29.342910 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:29.342873 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9662ce2_6f58_4385_8f20_f9ed018fea61.slice/crio-36fdfbd3e39437309b5783cfe1676fb681d59ff75d539f3a74e983fd199dae7f WatchSource:0}: Error finding container 36fdfbd3e39437309b5783cfe1676fb681d59ff75d539f3a74e983fd199dae7f: Status 404 returned error can't find the container with id 36fdfbd3e39437309b5783cfe1676fb681d59ff75d539f3a74e983fd199dae7f Nov 01 15:47:29.343503 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:29.343458 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d4d845_68cc_44e8_9503_5969ed5fcc1b.slice/crio-a762abb23c5ff323d4eca7f380e7ed7ef7d9f1803db31837f1c7a0ef8edf8c33 WatchSource:0}: Error finding container a762abb23c5ff323d4eca7f380e7ed7ef7d9f1803db31837f1c7a0ef8edf8c33: Status 404 returned error can't find the container with id a762abb23c5ff323d4eca7f380e7ed7ef7d9f1803db31837f1c7a0ef8edf8c33 Nov 01 15:47:29.406289 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.400034 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv"] Nov 01 15:47:29.409907 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.409293 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5qhqp"] Nov 01 15:47:29.412514 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:29.412490 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea9e9fb_3362_48cf_b2ec_3db89f981464.slice/crio-883aac2f75972cbd239bde86626d7aa6d14a591b4033e01619f1ca690e03126b WatchSource:0}: Error finding container 883aac2f75972cbd239bde86626d7aa6d14a591b4033e01619f1ca690e03126b: Status 404 returned error can't find the container with id 883aac2f75972cbd239bde86626d7aa6d14a591b4033e01619f1ca690e03126b Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.423333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.423381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.423413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.423454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.423527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.423600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.423657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.423790 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.423841 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.423822243 +0000 UTC m=+32.054542494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.423904 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.423938 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls podName:f70b67b3-d8eb-4065-8650-b26de9bc9f7c nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.423926802 +0000 UTC m=+32.054647050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5rr7" (UID: "f70b67b3-d8eb-4065-8650-b26de9bc9f7c") : secret "samples-operator-tls" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424002 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424013 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424059 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.424032637 +0000 UTC m=+32.054752882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424113 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:29.424419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424148 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls podName:57440ce3-722e-4835-927a-e8f2fac8b3c7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.424137499 +0000 UTC m=+32.054857744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-59xc9" (UID: "57440ce3-722e-4835-927a-e8f2fac8b3c7") : secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:29.425380 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424221 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.424209407 +0000 UTC m=+32.054929655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:29.425380 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424279 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 15:47:29.425380 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424306 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert podName:19f38b5c-f14d-48fd-b0cf-a2581c62384a nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.424298021 +0000 UTC m=+32.055018269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j46fh" (UID: "19f38b5c-f14d-48fd-b0cf-a2581c62384a") : secret "networking-console-plugin-cert" not found Nov 01 15:47:29.425380 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424356 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:29.425380 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424364 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:29.425380 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.424386 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.424379698 +0000 UTC m=+32.055099943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:29.442654 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.442625 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99"] Nov 01 15:47:29.466697 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.466671 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj"] Nov 01 15:47:29.469977 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:29.469953 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adcbe8f_dcd5_47d5_bd8a_68f0206a7d04.slice/crio-9b5fc71dfd363f6bafc11949156b7d6c211fdcd3e48869d87052b244572aee24 WatchSource:0}: Error finding container 9b5fc71dfd363f6bafc11949156b7d6c211fdcd3e48869d87052b244572aee24: Status 404 returned error can't find the container with id 9b5fc71dfd363f6bafc11949156b7d6c211fdcd3e48869d87052b244572aee24 Nov 01 15:47:29.470249 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.470220 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fz7xr"] Nov 01 15:47:29.472560 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:29.472537 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a31617_3eb9_499e_a192_f4d24a34e835.slice/crio-869c87fc97a4e70718c6ce199b5f1dce0857043325eeacf13421ce2ff0914464 WatchSource:0}: Error finding container 869c87fc97a4e70718c6ce199b5f1dce0857043325eeacf13421ce2ff0914464: Status 404 returned error can't find the container with id 869c87fc97a4e70718c6ce199b5f1dce0857043325eeacf13421ce2ff0914464 Nov 01 15:47:29.521719 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.521704 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cf2mk"] Nov 01 15:47:29.523403 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:29.523375 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6d982a_e0d9_42ee_b342_290a02120b36.slice/crio-bc9cc25584a67745436631926c54e00ee56c57b1a8816271364ef4df4e6a2ada WatchSource:0}: Error finding container bc9cc25584a67745436631926c54e00ee56c57b1a8816271364ef4df4e6a2ada: Status 404 returned error can't find the container with id bc9cc25584a67745436631926c54e00ee56c57b1a8816271364ef4df4e6a2ada Nov 01 15:47:29.524441 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.524417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:29.524569 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.524556 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:29.524613 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:29.524607 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:30.52459297 +0000 UTC m=+32.155313216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:29.905769 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.905676 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:29.905769 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.905724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:29.905963 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.905793 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:29.909337 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.909312 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2bqxm\"" Nov 01 15:47:29.909744 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.909383 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qm9xj\"" Nov 01 15:47:29.909744 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.909441 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 15:47:29.909744 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.909460 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 15:47:29.909744 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:29.909492 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4p8cp\"" Nov 01 15:47:30.165716 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.165555 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" event={"ID":"4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04","Type":"ContainerStarted","Data":"9b5fc71dfd363f6bafc11949156b7d6c211fdcd3e48869d87052b244572aee24"} Nov 01 15:47:30.170700 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.170615 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" event={"ID":"8a515047-c16a-4352-acaf-16b5bd0dea55","Type":"ContainerStarted","Data":"a5c3860132f0930f0440ed7fecbf576abe29003d8dc1f21d98aa86a0f3118838"} Nov 01 15:47:30.176662 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.175893 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" event={"ID":"85d4d845-68cc-44e8-9503-5969ed5fcc1b","Type":"ContainerStarted","Data":"c40d62462acf45148d247dd47c710b1693cc4bc812aba2b6b66c8524bc531a87"} Nov 01 15:47:30.176662 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.175948 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" event={"ID":"85d4d845-68cc-44e8-9503-5969ed5fcc1b","Type":"ContainerStarted","Data":"a762abb23c5ff323d4eca7f380e7ed7ef7d9f1803db31837f1c7a0ef8edf8c33"} Nov 01 15:47:30.180070 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.179358 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz7xr" event={"ID":"23a31617-3eb9-499e-a192-f4d24a34e835","Type":"ContainerStarted","Data":"869c87fc97a4e70718c6ce199b5f1dce0857043325eeacf13421ce2ff0914464"} Nov 01 15:47:30.182686 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.182659 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" event={"ID":"cea9e9fb-3362-48cf-b2ec-3db89f981464","Type":"ContainerStarted","Data":"883aac2f75972cbd239bde86626d7aa6d14a591b4033e01619f1ca690e03126b"} Nov 01 15:47:30.185633 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.185563 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" event={"ID":"f1f800e1-24b3-4323-97f3-2009efb63d1b","Type":"ContainerStarted","Data":"ebc811ca5f4ba37651300825ae8721966cfb27a227e21d6a99b0eb86b18d70a8"} Nov 01 15:47:30.190209 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.190154 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" event={"ID":"a9662ce2-6f58-4385-8f20-f9ed018fea61","Type":"ContainerStarted","Data":"36fdfbd3e39437309b5783cfe1676fb681d59ff75d539f3a74e983fd199dae7f"} Nov 01 15:47:30.194393 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.192944 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" podStartSLOduration=421.192928891 podStartE2EDuration="7m1.192928891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:30.192509718 +0000 UTC m=+31.823229989" watchObservedRunningTime="2026-11-01 15:47:30.192928891 +0000 UTC m=+31.823649162" Nov 01 15:47:30.200086 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.199602 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" event={"ID":"3be78730-9048-43e9-9d31-e3819027d095","Type":"ContainerStarted","Data":"19de63d70702447c0944875d8d5ef91cc50cbf119974a43497fdc123fd0bad6b"} Nov 01 15:47:30.200086 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.199645 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" event={"ID":"3be78730-9048-43e9-9d31-e3819027d095","Type":"ContainerStarted","Data":"29210824966f85f75f01690b769716d5e8887c2b8f87d4f2c4082b717e52b402"} Nov 01 15:47:30.203782 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.203723 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cf2mk" event={"ID":"8f6d982a-e0d9-42ee-b342-290a02120b36","Type":"ContainerStarted","Data":"bc9cc25584a67745436631926c54e00ee56c57b1a8816271364ef4df4e6a2ada"} Nov 01 15:47:30.212064 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.211557 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" event={"ID":"1a7cd5bd-620b-4d98-b64a-4582788161a4","Type":"ContainerStarted","Data":"bd4adee2a5d1f68c9092c41af5ff46922f8d7edcfe81834bad96c9f60bfc69b3"} Nov 01 15:47:30.212064 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.211599 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" event={"ID":"1a7cd5bd-620b-4d98-b64a-4582788161a4","Type":"ContainerStarted","Data":"a56f9cfa54ee501e104549baa9964260c9ce82087d7d9197157ffc5284c84180"} Nov 01 15:47:30.234592 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.232919 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" podStartSLOduration=421.232903922 podStartE2EDuration="7m1.232903922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:30.232630253 +0000 UTC m=+31.863350533" watchObservedRunningTime="2026-11-01 15:47:30.232903922 +0000 UTC m=+31.863624190" Nov 01 15:47:30.234592 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.233347 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" podStartSLOduration=421.233336086 podStartE2EDuration="7m1.233336086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:30.216259946 +0000 UTC m=+31.846980214" watchObservedRunningTime="2026-11-01 15:47:30.233336086 +0000 UTC m=+31.864056361" Nov 01 15:47:30.434190 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.434103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:30.434359 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.434202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:30.434359 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.434249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:30.434359 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.434282 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:30.434359 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.434310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:30.434359 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.434348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:30.434605 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.434414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:30.434605 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.434589 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 15:47:30.434703 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.434652 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert podName:19f38b5c-f14d-48fd-b0cf-a2581c62384a nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.434634549 +0000 UTC m=+34.065354811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j46fh" (UID: "19f38b5c-f14d-48fd-b0cf-a2581c62384a") : secret "networking-console-plugin-cert" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435073 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435092 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435138 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.435123329 +0000 UTC m=+34.065843589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435203 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435238 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.435225755 +0000 UTC m=+34.065946015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435296 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435326 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls podName:f70b67b3-d8eb-4065-8650-b26de9bc9f7c nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.43531618 +0000 UTC m=+34.066036438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5rr7" (UID: "f70b67b3-d8eb-4065-8650-b26de9bc9f7c") : secret "samples-operator-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435381 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435389 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435417 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.435407769 +0000 UTC m=+34.066128032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435479 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435510 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls podName:57440ce3-722e-4835-927a-e8f2fac8b3c7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.435500756 +0000 UTC m=+34.066221015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-59xc9" (UID: "57440ce3-722e-4835-927a-e8f2fac8b3c7") : secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:30.435614 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.435574 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.435564478 +0000 UTC m=+34.066284737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:30.535923 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:30.535268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:30.535923 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.535520 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:30.535923 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:30.535577 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.535561327 +0000 UTC m=+34.166281586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:32.226516 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.226441 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" event={"ID":"f1f800e1-24b3-4323-97f3-2009efb63d1b","Type":"ContainerStarted","Data":"14cd3900eefaba535ada710312b37cdb55c559ae15c1454cb51f8e57e5a850e2"} Nov 01 15:47:32.246627 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.246158 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" podStartSLOduration=390.502580617 podStartE2EDuration="6m32.246138977s" totalImagesPullingTime="1.74355836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:32.245146273 +0000 UTC m=+33.875866540" watchObservedRunningTime="2026-11-01 15:47:32.246138977 +0000 UTC m=+33.876859244" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.457392 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.457497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.457532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.457562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.457601 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457598 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.457668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457709 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.457686782 +0000 UTC m=+38.088407042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457761 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.457782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457839 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert podName:19f38b5c-f14d-48fd-b0cf-a2581c62384a nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.457824622 +0000 UTC m=+38.088544871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j46fh" (UID: "19f38b5c-f14d-48fd-b0cf-a2581c62384a") : secret "networking-console-plugin-cert" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457904 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457917 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457918 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457930 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:32.458269 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457949 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.45793827 +0000 UTC m=+38.088658531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:32.459553 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457964 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.457956262 +0000 UTC m=+38.088676512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:32.459553 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.457996 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:32.459553 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.458028 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls podName:57440ce3-722e-4835-927a-e8f2fac8b3c7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.458017103 +0000 UTC m=+38.088737351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-59xc9" (UID: "57440ce3-722e-4835-927a-e8f2fac8b3c7") : secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:32.459553 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.458102 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.45809104 +0000 UTC m=+38.088811287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:32.459553 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.458138 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 15:47:32.459553 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.458169 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls podName:f70b67b3-d8eb-4065-8650-b26de9bc9f7c nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.458156918 +0000 UTC m=+38.088877166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5rr7" (UID: "f70b67b3-d8eb-4065-8650-b26de9bc9f7c") : secret "samples-operator-tls" not found Nov 01 15:47:32.558784 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.558742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:32.558942 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.558865 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:32.558942 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.558874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:32.558942 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.558905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:32.558942 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.558930 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.558912095 +0000 UTC m=+38.189632353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:32.559419 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.559394 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 15:47:32.559498 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:32.559453 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs podName:8b24f7e8-b574-4609-96e5-9908302ac283 nodeName:}" failed. No retries permitted until 2026-11-01 15:48:04.559438446 +0000 UTC m=+66.190158705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs") pod "network-metrics-daemon-l6qq5" (UID: "8b24f7e8-b574-4609-96e5-9908302ac283") : secret "metrics-daemon-secret" not found Nov 01 15:47:32.571512 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.571489 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:32.659632 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.659599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:32.666628 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.666571 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:32.673123 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.673096 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:32.964960 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:32.964612 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:33.034565 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:33.034120 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xvq6f"] Nov 01 15:47:33.223510 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:33.223426 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9337a97_e8fa_4061_ba5f_fea449c7d390.slice/crio-7423d5afff283d04bc8ad7e0c1eace43c70347870211451fc6cd8d48f6993863 WatchSource:0}: Error finding container 7423d5afff283d04bc8ad7e0c1eace43c70347870211451fc6cd8d48f6993863: Status 404 returned error can't find the container with id 7423d5afff283d04bc8ad7e0c1eace43c70347870211451fc6cd8d48f6993863 Nov 01 15:47:33.232545 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:33.232481 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xvq6f" event={"ID":"e9337a97-e8fa-4061-ba5f-fea449c7d390","Type":"ContainerStarted","Data":"7423d5afff283d04bc8ad7e0c1eace43c70347870211451fc6cd8d48f6993863"} Nov 01 15:47:33.253077 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:33.253052 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdfqx"] Nov 01 15:47:33.258244 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:33.258221 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cf87aa_e08e_4890_acf5_c8da31cd08da.slice/crio-9578d6292af38dda6b140d751816070e36652fe4e62e1cd9d0c2581f1c2ce3d2 WatchSource:0}: Error finding container 9578d6292af38dda6b140d751816070e36652fe4e62e1cd9d0c2581f1c2ce3d2: Status 404 returned error can't find the container with id 9578d6292af38dda6b140d751816070e36652fe4e62e1cd9d0c2581f1c2ce3d2 Nov 01 15:47:34.236857 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:34.236828 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/0.log" Nov 01 15:47:34.237223 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:34.236865 2570 generic.go:356] "Generic (PLEG): container finished" podID="cea9e9fb-3362-48cf-b2ec-3db89f981464" containerID="37e223074e9116acd9476abbe2b56f5b6ff45ed5aac7885250aec974e0212ac7" exitCode=255 Nov 01 15:47:34.237223 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:34.236953 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" event={"ID":"cea9e9fb-3362-48cf-b2ec-3db89f981464","Type":"ContainerDied","Data":"37e223074e9116acd9476abbe2b56f5b6ff45ed5aac7885250aec974e0212ac7"} Nov 01 15:47:34.237223 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:34.237191 2570 scope.go:122] "RemoveContainer" containerID="37e223074e9116acd9476abbe2b56f5b6ff45ed5aac7885250aec974e0212ac7" Nov 01 15:47:34.238512 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:34.238476 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdfqx" event={"ID":"09cf87aa-e08e-4890-acf5-c8da31cd08da","Type":"ContainerStarted","Data":"9578d6292af38dda6b140d751816070e36652fe4e62e1cd9d0c2581f1c2ce3d2"} Nov 01 15:47:35.046537 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.046513 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6x8kz"] Nov 01 15:47:35.076165 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.073560 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6x8kz"] Nov 01 15:47:35.076165 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.073715 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.078307 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.078285 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtfck\"" Nov 01 15:47:35.078502 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.078482 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 15:47:35.109292 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.109265 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 15:47:35.190084 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.186814 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b220d197-999f-4b20-af10-f9ffb9420614-data-volume\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.190084 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.186917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.190084 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.186961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9k9p\" (UniqueName: \"kubernetes.io/projected/b220d197-999f-4b20-af10-f9ffb9420614-kube-api-access-c9k9p\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.190084 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.186997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b220d197-999f-4b20-af10-f9ffb9420614-crio-socket\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.190084 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.187102 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.251076 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.251013 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cf2mk" event={"ID":"8f6d982a-e0d9-42ee-b342-290a02120b36","Type":"ContainerStarted","Data":"7e6c61d714cd5c1fddfabd8ef9b2167839f0ab9ac1b56764e7a4f869b5fcba19"} Nov 01 15:47:35.253545 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.253518 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" event={"ID":"4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04","Type":"ContainerStarted","Data":"0fd0a4ff9d38fca559d8f29cf5fe24a180cfd2e18ec17858529432941e31564c"} Nov 01 15:47:35.255929 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.255802 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" event={"ID":"8a515047-c16a-4352-acaf-16b5bd0dea55","Type":"ContainerStarted","Data":"86c05e813218aeb00b8629ba3257f94b0de50f1181136f6a6d6d2161f00057b5"} Nov 01 15:47:35.258799 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.258281 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz7xr" event={"ID":"23a31617-3eb9-499e-a192-f4d24a34e835","Type":"ContainerStarted","Data":"650587bc6b5d8964e5286a66e4a610872a463f6d2fa2e98aeb4111a558dcfe10"} Nov 01 15:47:35.260820 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.260801 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/1.log" Nov 01 15:47:35.262321 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.262282 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/0.log" Nov 01 15:47:35.262401 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.262346 2570 generic.go:356] "Generic (PLEG): container finished" podID="cea9e9fb-3362-48cf-b2ec-3db89f981464" containerID="f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66" exitCode=255 Nov 01 15:47:35.262457 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.262436 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" event={"ID":"cea9e9fb-3362-48cf-b2ec-3db89f981464","Type":"ContainerDied","Data":"f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66"} Nov 01 15:47:35.262522 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.262477 2570 scope.go:122] "RemoveContainer" containerID="37e223074e9116acd9476abbe2b56f5b6ff45ed5aac7885250aec974e0212ac7" Nov 01 15:47:35.262648 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.262611 2570 scope.go:122] "RemoveContainer" containerID="f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66" Nov 01 15:47:35.262832 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.262802 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:35.265519 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.265188 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" event={"ID":"a9662ce2-6f58-4385-8f20-f9ed018fea61","Type":"ContainerStarted","Data":"96dc33b92346765a29b92d9c11d93d7a2442495ab7398c53ac0f9700f7697002"} Nov 01 15:47:35.270116 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.270078 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cf2mk" podStartSLOduration=2.141642992 podStartE2EDuration="7.27006819s" totalImagesPullingTime="5.128425198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:35.268702673 +0000 UTC m=+36.899422939" watchObservedRunningTime="2026-11-01 15:47:35.27006819 +0000 UTC m=+36.900788457" Nov 01 15:47:35.290689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.288064 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b220d197-999f-4b20-af10-f9ffb9420614-data-volume\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.290689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.288207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.290689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.288298 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k9p\" (UniqueName: \"kubernetes.io/projected/b220d197-999f-4b20-af10-f9ffb9420614-kube-api-access-c9k9p\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.290689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.288355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b220d197-999f-4b20-af10-f9ffb9420614-crio-socket\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.290689 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.288518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.290689 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.288818 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:35.290689 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.288873 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:35.788855431 +0000 UTC m=+37.419575690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : configmap "kube-rbac-proxy" not found Nov 01 15:47:35.291961 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.291513 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b220d197-999f-4b20-af10-f9ffb9420614-crio-socket\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.291961 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.291536 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.291961 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.291588 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:35.791575813 +0000 UTC m=+37.422296061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.292525 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.292480 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b220d197-999f-4b20-af10-f9ffb9420614-data-volume\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.312279 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.311087 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" podStartSLOduration=386.543382496 podStartE2EDuration="6m31.311070274s" totalImagesPullingTime="4.767687778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:35.285696982 +0000 UTC m=+36.916417251" watchObservedRunningTime="2026-11-01 15:47:35.311070274 +0000 UTC m=+36.941790545" Nov 01 15:47:35.317890 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.317735 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k9p\" (UniqueName: \"kubernetes.io/projected/b220d197-999f-4b20-af10-f9ffb9420614-kube-api-access-c9k9p\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.336400 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.336346 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-fz7xr" podStartSLOduration=384.602785754 podStartE2EDuration="6m29.336326936s" totalImagesPullingTime="4.733541182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:35.333803467 +0000 UTC m=+36.964523747" watchObservedRunningTime="2026-11-01 15:47:35.336326936 +0000 UTC m=+36.967047204" Nov 01 15:47:35.350599 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.350533 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" podStartSLOduration=388.843129441 podStartE2EDuration="6m34.350515149s" totalImagesPullingTime="5.507385708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:35.348348666 +0000 UTC m=+36.979068935" watchObservedRunningTime="2026-11-01 15:47:35.350515149 +0000 UTC m=+36.981235418" Nov 01 15:47:35.372082 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.371752 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" podStartSLOduration=390.509018334 podStartE2EDuration="6m35.371732821s" totalImagesPullingTime="4.862714487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:35.370398339 +0000 UTC m=+37.001118607" watchObservedRunningTime="2026-11-01 15:47:35.371732821 +0000 UTC m=+37.002453090" Nov 01 15:47:35.562570 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.561209 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx"] Nov 01 15:47:35.608182 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.608145 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx"] Nov 01 15:47:35.608182 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.608167 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" Nov 01 15:47:35.678329 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.678296 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:35.678329 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.678333 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4c2bg\"" Nov 01 15:47:35.678551 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.678387 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 15:47:35.792593 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.792552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.792754 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.792644 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzh9b\" (UniqueName: \"kubernetes.io/projected/5671cc22-cecc-4d28-9054-9eee4e8dae58-kube-api-access-pzh9b\") pod \"migrator-66df5d496b-mz2hx\" (UID: \"5671cc22-cecc-4d28-9054-9eee4e8dae58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" Nov 01 15:47:35.792754 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.792734 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.792754 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.792742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:35.792922 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.792803 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.79278331 +0000 UTC m=+38.423503560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.792922 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.792831 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:35.792922 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:35.792908 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.792889327 +0000 UTC m=+38.423609587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : configmap "kube-rbac-proxy" not found Nov 01 15:47:35.894159 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.893940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzh9b\" (UniqueName: \"kubernetes.io/projected/5671cc22-cecc-4d28-9054-9eee4e8dae58-kube-api-access-pzh9b\") pod \"migrator-66df5d496b-mz2hx\" (UID: \"5671cc22-cecc-4d28-9054-9eee4e8dae58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" Nov 01 15:47:35.906802 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.906768 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzh9b\" (UniqueName: \"kubernetes.io/projected/5671cc22-cecc-4d28-9054-9eee4e8dae58-kube-api-access-pzh9b\") pod \"migrator-66df5d496b-mz2hx\" (UID: \"5671cc22-cecc-4d28-9054-9eee4e8dae58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" Nov 01 15:47:35.928552 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:35.928522 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" Nov 01 15:47:36.226723 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.226694 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx"] Nov 01 15:47:36.230300 ip-10-0-143-101 kubenswrapper[2570]: W1101 15:47:36.230274 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5671cc22_cecc_4d28_9054_9eee4e8dae58.slice/crio-f05b349f0fc079a118d6224026e3e1e2c0fdb353b5aa8c8f8668f2af89230e83 WatchSource:0}: Error finding container f05b349f0fc079a118d6224026e3e1e2c0fdb353b5aa8c8f8668f2af89230e83: Status 404 returned error can't find the container with id f05b349f0fc079a118d6224026e3e1e2c0fdb353b5aa8c8f8668f2af89230e83 Nov 01 15:47:36.270612 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.270572 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" event={"ID":"5671cc22-cecc-4d28-9054-9eee4e8dae58","Type":"ContainerStarted","Data":"f05b349f0fc079a118d6224026e3e1e2c0fdb353b5aa8c8f8668f2af89230e83"} Nov 01 15:47:36.272564 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.272538 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/1.log" Nov 01 15:47:36.274111 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.273601 2570 scope.go:122] "RemoveContainer" containerID="f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66" Nov 01 15:47:36.274111 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.273758 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:36.504749 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.504708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:36.504958 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.504766 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:36.504958 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.504812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:36.504958 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.504890 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:36.504958 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.504915 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:36.504958 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.504955 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.50493732 +0000 UTC m=+46.135657579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.504971 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.504978 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.504967136 +0000 UTC m=+46.135687381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505056 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls podName:57440ce3-722e-4835-927a-e8f2fac8b3c7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.505024098 +0000 UTC m=+46.135744350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-59xc9" (UID: "57440ce3-722e-4835-927a-e8f2fac8b3c7") : secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.505088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.505151 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505212 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.505227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505276 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert podName:19f38b5c-f14d-48fd-b0cf-a2581c62384a nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.505260809 +0000 UTC m=+46.135981068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j46fh" (UID: "19f38b5c-f14d-48fd-b0cf-a2581c62384a") : secret "networking-console-plugin-cert" not found Nov 01 15:47:36.505294 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505288 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:36.505634 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505306 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:36.505634 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505308 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:36.505634 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.505301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:36.505634 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505350 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.505333943 +0000 UTC m=+46.136054192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:36.505634 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505389 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 15:47:36.505634 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505420 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.505409473 +0000 UTC m=+46.136129722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:36.505634 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.505437 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls podName:f70b67b3-d8eb-4065-8650-b26de9bc9f7c nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.505428502 +0000 UTC m=+46.136148821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5rr7" (UID: "f70b67b3-d8eb-4065-8650-b26de9bc9f7c") : secret "samples-operator-tls" not found Nov 01 15:47:36.606540 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.606503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:36.606711 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.606669 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:36.606766 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.606745 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.606725852 +0000 UTC m=+46.237446102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:36.809097 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.809072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:36.809642 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.809219 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:36.809642 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:36.809247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:36.809642 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.809300 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:38.809280288 +0000 UTC m=+40.440000535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : configmap "kube-rbac-proxy" not found Nov 01 15:47:36.809642 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.809379 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:36.809642 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:36.809421 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:38.809407554 +0000 UTC m=+40.440127800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:37.279556 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:37.279516 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdfqx" event={"ID":"09cf87aa-e08e-4890-acf5-c8da31cd08da","Type":"ContainerStarted","Data":"7dd3a14847d0bf9a5e4edb7ef0c86cdc7f54f7b7d086656b231e97fa7f45fd28"} Nov 01 15:47:37.280134 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:37.279686 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:38.284990 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:38.284950 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" event={"ID":"5671cc22-cecc-4d28-9054-9eee4e8dae58","Type":"ContainerStarted","Data":"8e9a9a7fbe0d5cecc3873c3d9a205b0c75b5f96e53384a3bdadb319387fc2786"} Nov 01 15:47:38.300783 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:38.300738 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" podStartSLOduration=1.6821162570000001 podStartE2EDuration="3.300724699s" totalImagesPullingTime="1.618608442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:38.300299019 +0000 UTC m=+39.931019283" watchObservedRunningTime="2026-11-01 15:47:38.300724699 +0000 UTC m=+39.931444965" Nov 01 15:47:38.301289 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:38.301260 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bdfqx" podStartSLOduration=35.821729553 podStartE2EDuration="39.301252173s" totalImagesPullingTime="3.47952262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:37.296474538 +0000 UTC m=+38.927194806" watchObservedRunningTime="2026-11-01 15:47:38.301252173 +0000 UTC m=+39.931972469" Nov 01 15:47:38.830067 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:38.830009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:38.830246 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:38.830141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:38.830246 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:38.830175 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:38.830352 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:38.830254 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.83023399 +0000 UTC m=+44.460954237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:38.830352 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:38.830255 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:38.830352 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:38.830297 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.830286262 +0000 UTC m=+44.461006510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : configmap "kube-rbac-proxy" not found Nov 01 15:47:39.052286 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:39.052258 2570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:39.052431 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:39.052318 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:39.052655 ip-10-0-143-101 kubenswrapper[2570]: I1101 15:47:39.052643 2570 scope.go:122] "RemoveContainer" containerID="f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66" Nov 01 15:47:39.052834 ip-10-0-143-101 kubenswrapper[2570]: E1101 15:47:39.052816 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:39.800964 ip-10-0-143-101 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 15:47:39.810451 ip-10-0-143-101 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 15:47:39.810629 ip-10-0-143-101 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 15:47:39.810680 ip-10-0-143-101 systemd[1]: kubelet.service: Consumed 3.048s CPU time. Nov 01 15:47:39.825836 ip-10-0-143-101 systemd[1]: Starting Kubernetes Kubelet... Nov 01 15:47:39.884335 ip-10-0-143-101 kubenswrapper[4911]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:39.884335 ip-10-0-143-101 kubenswrapper[4911]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 15:47:39.884335 ip-10-0-143-101 kubenswrapper[4911]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:39.884335 ip-10-0-143-101 kubenswrapper[4911]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:39.886375 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886282 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.886375 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886368 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.886375 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886372 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.886375 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886375 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.886375 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886378 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.886375 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886381 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.886375 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886384 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886388 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886391 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886394 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886397 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886399 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886402 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886405 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886408 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886410 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886413 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886415 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886418 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886426 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886429 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886432 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886436 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886441 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886445 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886449 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.886730 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886454 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886458 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886462 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886466 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886470 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886474 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886478 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886481 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886485 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886490 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886494 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886498 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886502 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886506 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886510 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886514 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886517 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886521 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886525 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886529 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.887287 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886533 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886536 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886541 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886545 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886550 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886554 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886558 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886562 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886566 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886570 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886574 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886577 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886581 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886585 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886589 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886593 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886600 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886606 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886610 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.887801 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886614 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886619 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886622 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886626 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886630 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886635 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886640 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886644 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886648 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886652 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886656 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886660 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886664 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886668 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886672 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886676 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886680 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886686 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886692 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.888299 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886697 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886702 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886705 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886708 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886711 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886713 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886716 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886719 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886722 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886724 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886727 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886729 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886732 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886735 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886738 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886741 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886743 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886746 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886749 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886753 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.888821 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886756 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.886758 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887581 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887590 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887593 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887596 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887599 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887602 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887605 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887607 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887610 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887613 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887615 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887618 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887620 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887623 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887626 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887628 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887631 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.889370 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887636 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887640 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887643 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887646 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887650 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887653 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887655 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887659 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887662 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887665 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887668 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887671 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887673 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887677 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887679 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887683 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887685 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887688 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887691 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887693 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.889876 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887700 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887703 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887706 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887709 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887712 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887714 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887717 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887719 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887722 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887724 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887727 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887730 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887733 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887736 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887738 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887741 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887744 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887746 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887750 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887753 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.890420 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887756 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887758 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887761 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887764 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887766 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887769 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887772 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887775 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887777 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887780 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887782 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887785 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887789 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887792 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887795 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887797 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887800 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887802 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887805 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887808 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.890993 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887812 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887815 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887817 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887820 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887823 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887826 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887828 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887831 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887833 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887836 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887839 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887841 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887844 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887846 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887849 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887852 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887855 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887857 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887860 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.891521 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887863 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887866 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887868 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887871 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887874 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887878 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887882 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887885 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887887 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.887890 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888387 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888393 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888396 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888399 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888402 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888404 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888407 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888410 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888413 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.891998 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888415 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888418 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888421 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888423 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888426 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888429 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888431 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888434 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888436 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888439 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888441 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888444 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888447 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888450 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888453 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888456 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888458 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888461 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888464 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888467 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.892497 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888469 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888472 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888475 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888477 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888480 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888482 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888485 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888487 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888490 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888492 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888495 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888497 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888500 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888503 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888506 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888509 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888511 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888514 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888516 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888518 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.893223 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888521 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888524 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888527 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888529 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888532 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888534 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888537 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888540 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888545 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888549 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888552 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888555 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888558 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888560 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888563 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888566 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888568 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888571 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888573 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.893964 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888576 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888578 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888581 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888584 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888587 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888589 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888592 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888594 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888597 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888599 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888601 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888604 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888607 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888609 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888612 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888614 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888617 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888619 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888622 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888624 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.894466 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888628 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888630 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888633 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888635 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888638 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888640 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888644 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888649 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888651 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888654 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888657 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888659 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888662 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888664 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888667 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888669 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888672 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.888674 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888767 4911 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 15:47:39.894976 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888774 4911 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888781 4911 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888785 4911 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888790 4911 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888793 4911 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888797 4911 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888802 4911 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888805 4911 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888808 4911 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888811 4911 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888815 4911 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888818 4911 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888821 4911 flags.go:64] FLAG: --cgroup-root="" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888824 4911 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888827 4911 flags.go:64] FLAG: --client-ca-file="" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888830 4911 flags.go:64] FLAG: --cloud-provider="external" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888834 4911 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888838 4911 flags.go:64] FLAG: --cluster-domain="" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888841 4911 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888846 4911 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888849 4911 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888853 4911 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888857 4911 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 15:47:39.895472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888860 4911 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888863 4911 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888866 4911 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888870 4911 flags.go:64] FLAG: --contention-profiling="false" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888873 4911 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888877 4911 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888880 4911 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888883 4911 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888888 4911 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888891 4911 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888894 4911 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888896 4911 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888900 4911 flags.go:64] FLAG: --enable-server="true" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888903 4911 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888907 4911 flags.go:64] FLAG: --event-burst="100" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888911 4911 flags.go:64] FLAG: --event-qps="50" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888914 4911 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888917 4911 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888920 4911 flags.go:64] FLAG: --eviction-hard="" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888925 4911 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888928 4911 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888931 4911 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888934 4911 flags.go:64] FLAG: --eviction-soft="" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888937 4911 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888941 4911 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 15:47:39.896079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888944 4911 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888947 4911 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888950 4911 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888954 4911 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888957 4911 flags.go:64] FLAG: --feature-gates="" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888961 4911 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888964 4911 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888968 4911 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888971 4911 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888974 4911 flags.go:64] FLAG: --healthz-port="10248" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888977 4911 flags.go:64] FLAG: --help="false" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888981 4911 flags.go:64] FLAG: --hostname-override="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888986 4911 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888989 4911 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888992 4911 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888995 4911 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.888999 4911 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889017 4911 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889020 4911 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889023 4911 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889027 4911 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889029 4911 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889033 4911 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889048 4911 flags.go:64] FLAG: --kube-reserved="" Nov 01 15:47:39.896743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889051 4911 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889054 4911 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889058 4911 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889060 4911 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889064 4911 flags.go:64] FLAG: --lock-file="" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889067 4911 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889070 4911 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889073 4911 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889090 4911 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889093 4911 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889096 4911 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889101 4911 flags.go:64] FLAG: --logging-format="text" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889104 4911 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889108 4911 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889111 4911 flags.go:64] FLAG: --manifest-url="" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889113 4911 flags.go:64] FLAG: --manifest-url-header="" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889118 4911 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889122 4911 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889125 4911 flags.go:64] FLAG: --max-pods="110" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889128 4911 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889136 4911 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889139 4911 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889142 4911 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889145 4911 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889148 4911 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 15:47:39.897334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889151 4911 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889158 4911 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889162 4911 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889165 4911 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889168 4911 flags.go:64] FLAG: --pod-cidr="" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889171 4911 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889174 4911 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889177 4911 flags.go:64] FLAG: --pods-per-core="0" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889180 4911 flags.go:64] FLAG: --port="10250" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889183 4911 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889186 4911 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-00d842a5f61b98997" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889190 4911 flags.go:64] FLAG: --qos-reserved="" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889193 4911 flags.go:64] FLAG: --read-only-port="10255" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889196 4911 flags.go:64] FLAG: --register-node="true" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889199 4911 flags.go:64] FLAG: --register-with-taints="" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889203 4911 flags.go:64] FLAG: --registry-burst="10" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889206 4911 flags.go:64] FLAG: --registry-qps="5" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889209 4911 flags.go:64] FLAG: --reserved-cpus="" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889213 4911 flags.go:64] FLAG: --reserved-memory="" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889217 4911 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889220 4911 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889223 4911 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889226 4911 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889229 4911 flags.go:64] FLAG: --runonce="false" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889232 4911 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889236 4911 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889239 4911 flags.go:64] FLAG: --seccomp-default="false" Nov 01 15:47:39.897943 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889243 4911 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889246 4911 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889249 4911 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889252 4911 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889255 4911 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889258 4911 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889261 4911 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889264 4911 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889267 4911 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889270 4911 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889273 4911 flags.go:64] FLAG: --system-cgroups="" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889276 4911 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889282 4911 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889284 4911 flags.go:64] FLAG: --tls-cert-file="" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889287 4911 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889291 4911 flags.go:64] FLAG: --tls-min-version="" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889294 4911 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889297 4911 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889300 4911 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889302 4911 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889305 4911 flags.go:64] FLAG: --v="2" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889310 4911 flags.go:64] FLAG: --version="false" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889315 4911 flags.go:64] FLAG: --vmodule="" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889320 4911 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889324 4911 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 15:47:39.898654 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889455 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889460 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889463 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889465 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889468 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889471 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889473 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889477 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889480 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889482 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889485 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889488 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889490 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889493 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889496 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889498 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889501 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889504 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889507 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.899301 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889509 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889512 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889515 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889518 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889520 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889523 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889526 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889528 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889531 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889533 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889536 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889540 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889543 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889545 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889548 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889551 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889553 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889556 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889559 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889561 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.899776 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889565 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889567 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889570 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889572 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889575 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889577 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889580 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889583 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889585 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889588 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889590 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889593 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889596 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889598 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889601 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889604 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889606 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889609 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889612 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.900341 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889615 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889617 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889620 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889623 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889628 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889632 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889635 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889638 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889641 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889643 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889646 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889648 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889651 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889655 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889658 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889660 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889663 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889665 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889668 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.900937 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889670 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889673 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889676 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889678 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889681 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889684 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889686 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889689 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889691 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889694 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889696 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889699 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889701 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889704 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889707 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889711 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889715 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889720 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889723 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.901552 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889727 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889730 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889733 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889735 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889738 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889741 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889743 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889747 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889750 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889752 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.889782 4911 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889843 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889848 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889851 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889854 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889857 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.902031 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889860 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889862 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889865 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889868 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889871 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889874 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889876 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889879 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889882 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889885 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889888 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889891 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889894 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889897 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889899 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889904 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889906 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889910 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889912 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.902492 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889915 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889919 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889922 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889925 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889929 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889932 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889934 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889937 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889941 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889943 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889947 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889949 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889952 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889955 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889957 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889960 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889962 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889965 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889967 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.903023 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889970 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889972 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889975 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889977 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889980 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889983 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889985 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889988 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889990 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889994 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889996 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.889999 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890004 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890007 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890010 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890013 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890016 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890020 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890023 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.903677 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890026 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890029 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890031 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890033 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890047 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890050 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890053 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890055 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890058 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890061 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890063 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890066 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890068 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890071 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890073 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890076 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890079 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890081 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890084 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.904196 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890086 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890089 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890091 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890095 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890098 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890101 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890103 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890107 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890109 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890112 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890115 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890119 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890122 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890125 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890127 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890130 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890133 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890135 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890138 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890141 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.904707 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890143 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890146 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890148 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890151 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.890154 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.895415 4911 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.895432 4911 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.895451 4911 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.895456 4911 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895523 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895529 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895533 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895537 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895540 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895544 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895547 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895549 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895552 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895555 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.905207 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895558 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895561 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895563 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895566 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895569 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895572 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895574 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895577 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895580 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895583 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895586 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895589 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895591 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895594 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895597 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895599 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895603 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895606 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895613 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.905675 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895616 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895619 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895621 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895624 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895627 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895630 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895633 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895635 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895638 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895640 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895643 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895646 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895648 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895652 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895654 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895657 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895660 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895662 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895665 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895669 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.906146 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895673 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895675 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895678 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895681 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895683 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895686 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895689 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895691 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895694 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895697 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895700 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895702 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895705 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895708 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895710 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895713 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895716 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895718 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895721 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895724 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.906753 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895726 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895729 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895732 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895735 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895738 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895741 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895743 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895746 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895749 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895752 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895755 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895757 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895760 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895762 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895765 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895768 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895770 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895773 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895776 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.907575 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895778 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895781 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895784 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895788 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895792 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895794 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895797 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895800 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895802 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895805 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895807 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895810 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895812 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895815 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895818 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895821 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895823 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.908255 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.895826 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.895854 4911 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896021 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896027 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896030 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896033 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896050 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896056 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896060 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896064 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896068 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896073 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896076 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896079 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896082 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896085 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.908670 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896088 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896090 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896093 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896096 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896099 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896102 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896104 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896107 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896109 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896112 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896115 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896117 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896120 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896123 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896125 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896128 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896131 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896133 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896136 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.909079 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896139 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896142 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896144 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896147 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896150 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896153 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896156 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896158 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896161 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896164 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896167 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896169 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896172 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896174 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896177 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896180 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896182 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896185 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896187 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896190 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.909539 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896193 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896195 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896198 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896201 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896203 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896206 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896208 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896211 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896213 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896217 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896221 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896224 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896227 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896229 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896232 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896234 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896238 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896240 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896244 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.910024 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896247 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896250 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896252 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896255 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896258 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896260 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896263 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896266 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896268 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896271 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896274 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896277 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896279 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896282 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896284 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896287 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896290 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896292 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896296 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.910564 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896300 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896304 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896306 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896309 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896312 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896315 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896317 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896320 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896322 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896325 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896328 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896330 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896333 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896336 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896339 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.896365 4911 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:39.911092 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896398 4911 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896403 4911 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896406 4911 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896409 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896412 4911 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896414 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896417 4911 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896420 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896423 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896425 4911 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896428 4911 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896430 4911 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896433 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896436 4911 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896439 4911 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896441 4911 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896444 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896447 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896449 4911 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:39.911505 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896452 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896455 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896457 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896460 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896463 4911 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896465 4911 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896468 4911 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896470 4911 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896473 4911 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896475 4911 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896478 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896481 4911 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896484 4911 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896486 4911 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896489 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896491 4911 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896494 4911 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896497 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896499 4911 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:39.911990 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896501 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896504 4911 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896507 4911 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896510 4911 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896512 4911 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896515 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896518 4911 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896520 4911 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896523 4911 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896525 4911 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896528 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896530 4911 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896533 4911 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896536 4911 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896538 4911 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896541 4911 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896544 4911 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896546 4911 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896549 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896551 4911 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:39.912603 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896554 4911 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896556 4911 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896560 4911 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896564 4911 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896567 4911 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896571 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896574 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896577 4911 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896580 4911 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896583 4911 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896585 4911 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896588 4911 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896591 4911 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896598 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896600 4911 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896603 4911 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896606 4911 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896608 4911 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896611 4911 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896614 4911 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:39.913150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896616 4911 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896619 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896622 4911 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896625 4911 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896628 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896630 4911 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896633 4911 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896635 4911 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896638 4911 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896641 4911 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896644 4911 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896648 4911 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896651 4911 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896653 4911 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896656 4911 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896659 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896661 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896664 4911 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896667 4911 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:39.913641 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896670 4911 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896672 4911 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896675 4911 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896678 4911 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896680 4911 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896683 4911 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896687 4911 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896691 4911 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:39.896694 4911 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.896768 4911 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.897807 4911 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.897863 4911 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.898059 4911 server.go:1014] "Starting client certificate rotation" Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.898148 4911 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.898193 4911 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 15:41:58 +0000 UTC" deadline="2028-07-05 01:07:32.080261594 +0000 UTC" Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.898227 4911 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14673h19m52.182036532s" Nov 01 15:47:39.914128 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.898559 4911 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:39.914559 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.899605 4911 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:39.914559 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.900949 4911 log.go:25] "Validated CRI v1 runtime API" Nov 01 15:47:39.914559 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.902635 4911 log.go:25] "Validated CRI v1 image API" Nov 01 15:47:39.914559 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.903370 4911 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 15:47:39.914559 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.905211 4911 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8eb0cbf0-b981-49cb-869e-8e9469ca800f:/dev/nvme0n1p4 f6ae0c86-9a61-4e20-b988-6449974d77f3:/dev/nvme0n1p3] Nov 01 15:47:39.914703 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.905226 4911 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29210824966f85f75f01690b769716d5e8887c2b8f87d4f2c4082b717e52b402/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29210824966f85f75f01690b769716d5e8887c2b8f87d4f2c4082b717e52b402/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36fdfbd3e39437309b5783cfe1676fb681d59ff75d539f3a74e983fd199dae7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36fdfbd3e39437309b5783cfe1676fb681d59ff75d539f3a74e983fd199dae7f/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5718d617383e9595df537f321da8012d69f7641236d0056d0fcc7e84c3ffa8c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5718d617383e9595df537f321da8012d69f7641236d0056d0fcc7e84c3ffa8c3/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6433c5acfa2b7f0a4a44db14196a2443c020c4f96ace892d41563527ac62e3f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6433c5acfa2b7f0a4a44db14196a2443c020c4f96ace892d41563527ac62e3f1/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6479c4becc1497c24f39a4338481f57c23b75a7cbdf2187c8035fd0128dc6a89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6479c4becc1497c24f39a4338481f57c23b75a7cbdf2187c8035fd0128dc6a89/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7423d5afff283d04bc8ad7e0c1eace43c70347870211451fc6cd8d48f6993863/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7423d5afff283d04bc8ad7e0c1eace43c70347870211451fc6cd8d48f6993863/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/869c87fc97a4e70718c6ce199b5f1dce0857043325eeacf13421ce2ff0914464/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/869c87fc97a4e70718c6ce199b5f1dce0857043325eeacf13421ce2ff0914464/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/883aac2f75972cbd239bde86626d7aa6d14a591b4033e01619f1ca690e03126b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/883aac2f75972cbd239bde86626d7aa6d14a591b4033e01619f1ca690e03126b/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9578d6292af38dda6b140d751816070e36652fe4e62e1cd9d0c2581f1c2ce3d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9578d6292af38dda6b140d751816070e36652fe4e62e1cd9d0c2581f1c2ce3d2/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b5fc71dfd363f6bafc11949156b7d6c211fdcd3e48869d87052b244572aee24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b5fc71dfd363f6bafc11949156b7d6c211fdcd3e48869d87052b244572aee24/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a56f9cfa54ee501e104549baa9964260c9ce82087d7d9197157ffc5284c84180/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a56f9cfa54ee501e104549baa9964260c9ce82087d7d9197157ffc5284c84180/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5c3860132f0930f0440ed7fecbf576abe29003d8dc1f21d98aa86a0f3118838/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5c3860132f0930f0440ed7fecbf576abe29003d8dc1f21d98aa86a0f3118838/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a762abb23c5ff323d4eca7f380e7ed7ef7d9f1803db31837f1c7a0ef8edf8c33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a762abb23c5ff323d4eca7f380e7ed7ef7d9f1803db31837f1c7a0ef8edf8c33/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb341de1665cba78f97eccebff37648402746cddbd8e6bce65b57f33ed6ecdfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb341de1665cba78f97eccebff37648402746cddbd8e6bce65b57f33ed6ecdfa/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc9cc25584a67745436631926c54e00ee56c57b1a8816271364ef4df4e6a2ada/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc9cc25584a67745436631926c54e00ee56c57b1a8816271364ef4df4e6a2ada/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e98087dbe24fa28c1d6fe98985df40ea9e3014555c2642a2e85c2885b79ccde9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e98087dbe24fa28c1d6fe98985df40ea9e3014555c2642a2e85c2885b79ccde9/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebc811ca5f4ba37651300825ae8721966cfb27a227e21d6a99b0eb86b18d70a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebc811ca5f4ba37651300825ae8721966cfb27a227e21d6a99b0eb86b18d70a8/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f05b349f0fc079a118d6224026e3e1e2c0fdb353b5aa8c8f8668f2af89230e83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f05b349f0fc079a118d6224026e3e1e2c0fdb353b5aa8c8f8668f2af89230e83/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f56ebf77df15b3f21f331172314cd8afccf9a4ed636db7491ff0591399f71cc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f56ebf77df15b3f21f331172314cd8afccf9a4ed636db7491ff0591399f71cc8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc84d1fcd7861299dd4b424fa3058fdd8bf0cbd0601ecda51f31e7a81fb84bc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc84d1fcd7861299dd4b424fa3058fdd8bf0cbd0601ecda51f31e7a81fb84bc9/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd9c772427de48f7c44ad67b56abd3ba59a0c928a6379a2c6772350b6e91c667/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd9c772427de48f7c44ad67b56abd3ba59a0c928a6379a2c6772350b6e91c667/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09cf87aa-e08e-4890-acf5-c8da31cd08da/volumes/kubernetes.io~projected/kube-api-access-t4vwg:{mountpoint:/var/lib/kubelet/pods/09cf87aa-e08e-4890-acf5-c8da31cd08da/volumes/kubernetes.io~projected/kube-api-access-t4vwg major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e085d33-5390-4d2e-a616-61160ee87e46/volumes/kubernetes.io~projected/kube-api-access-pw4mn:{mountpoint:/var/lib/kubelet/pods/0e085d33-5390-4d2e-a616-61160ee87e46/volumes/kubernetes.io~projected/kube-api-access-pw4mn major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a3d24bb-ebe0-4623-88d7-77cd90a8b142/volumes/kubernetes.io~projected/kube-api-access-gbwmh:{mountpoint:/var/lib/kubelet/pods/1a3d24bb-ebe0-4623-88d7-77cd90a8b142/volumes/kubernetes.io~projected/kube-api-access-gbwmh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a7cd5bd-620b-4d98-b64a-4582788161a4/volumes/kubernetes.io~projected/kube-api-access-rj4tt:{mountpoint:/var/lib/kubelet/pods/1a7cd5bd-620b-4d98-b64a-4582788161a4/volumes/kubernetes.io~projected/kube-api-access-rj4tt major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23a31617-3eb9-499e-a192-f4d24a34e835/volumes/kubernetes.io~projected/kube-api-access-gvm7b:{mountpoint:/var/lib/kubelet/pods/23a31617-3eb9-499e-a192-f4d24a34e835/volumes/kubernetes.io~projected/kube-api-access-gvm7b major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23a31617-3eb9-499e-a192-f4d24a34e835/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/23a31617-3eb9-499e-a192-f4d24a34e835/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~projected/kube-api-access-227hr:{mountpoint:/var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~projected/kube-api-access-227hr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3be78730-9048-43e9-9d31-e3819027d095/volumes/kubernetes.io~projected/kube-api-access-kqng2:{mountpoint:/var/lib/kubelet/pods/3be78730-9048-43e9-9d31-e3819027d095/volumes/kubernetes.io~projected/kube-api-access-kqng2 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~projected/kube-api-access-6rj4g:{mountpoint:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~projected/kube-api-access-6rj4g major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04/volumes/kubernetes.io~projected/kube-api-access-vsgh4:{mountpoint:/var/lib/kubelet/pods/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04/volumes/kubernetes.io~projected/kube-api-access-vsgh4 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~projected/kube-api-access-4m465:{mountpoint:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~projected/kube-api-access-4m465 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5671cc22-cecc-4d28-9054-9eee4e8dae58/volumes/kubernetes.io~projected/kube-api-access-pzh9b:{mountpoint:/var/lib/kubelet/pods/5671cc22-cecc-4d28-9054-9eee4e8dae58/volumes/kubernetes.io~projected/kube-api-access-pzh9b major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57440ce3-722e-4835-927a-e8f2fac8b3c7/volumes/kubernetes.io~projected/kube-api-access-p48zp:{mountpoint:/var/lib/kubelet/pods/57440ce3-722e-4835-927a-e8f2fac8b3c7/volumes/kubernetes.io~projected/kube-api-access-p48zp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7039a946-2ec8-414d-9093-7f355172fba4/volumes/kubernetes.io~projected/kube-api-access-x9vgf:{mountpoint:/var/lib/kubelet/pods/7039a946-2ec8-414d-9093-7f355172fba4/volumes/kubernetes.io~projected/kube-api-access-x9vgf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a901c1f-0f50-4d5e-94e8-03326a1c650f/volumes/kubernetes.io~projected/kube-api-access-rt658:{mountpoint:/var/lib/kubelet/pods/7a901c1f-0f50-4d5e-94e8-03326a1c650f/volumes/kubernetes.io~projected/kube-api-access-rt658 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85d4d845-68cc-44e8-9503-5969ed5fcc1b/volumes/kubernetes.io~projected/kube-api-access-zldzc:{mountpoint:/var/lib/kubelet/pods/85d4d845-68cc-44e8-9503-5969ed5fcc1b/volumes/kubernetes.io~projected/kube-api-access-zldzc major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a515047-c16a-4352-acaf-16b5bd0dea55/volumes/kubernetes.io~projected/kube-api-access-4qd88:{mountpoint:/var/lib/kubelet/pods/8a515047-c16a-4352-acaf-16b5bd0dea55/volumes/kubernetes.io~projected/kube-api-access-4qd88 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a515047-c16a-4352-acaf-16b5bd0dea55/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8a515047-c16a-4352-acaf-16b5bd0dea55/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b24f7e8-b574-4609-96e5-9908302ac283/volumes/kubernetes.io~projected/kube-api-access-j892r:{mountpoint:/var/lib/kubelet/pods/8b24f7e8-b574-4609-96e5-9908302ac283/volumes/kubernetes.io~projected/kube-api-access-j892r major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f6d982a-e0d9-42ee-b342-290a02120b36/volumes/kubernetes.io~projected/kube-api-access-598sq:{mountpoint:/var/lib/kubelet/pods/8f6d982a-e0d9-42ee-b342-290a02120b36/volumes/kubernetes.io~projected/kube-api-access-598sq major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f6d982a-e0d9-42ee-b342-290a02120b36/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8f6d982a-e0d9-42ee-b342-290a02120b36/volumes/kubernetes.io~secret/cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9662ce2-6f58-4385-8f20-f9ed018fea61/volumes/kubernetes.io~projected/kube-api-access-8vfx8:{mountpoint:/var/lib/kubelet/pods/a9662ce2-6f58-4385-8f20-f9ed018fea61/volumes/kubernetes.io~projected/kube-api-access-8vfx8 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9662ce2-6f58-4385-8f20-f9ed018fea61/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a9662ce2-6f58-4385-8f20-f9ed018fea61/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b220d197-999f-4b20-af10-f9ffb9420614/volumes/kubernetes.io~projected/kube-api-access-c9k9p:{mountpoint:/var/lib/kubelet/pods/b220d197-999f-4b20-af10-f9ffb9420614/volumes/kubernetes.io~projected/kube-api-access-c9k9p major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea9e9fb-3362-48cf-b2ec-3db89f981464/volumes/kubernetes.io~projected/kube-api-access-xmfn8:{mountpoint:/var/lib/kubelet/pods/cea9e9fb-3362-48cf-b2ec-3db89f981464/volumes/kubernetes.io~projected/kube-api-access-xmfn8 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cea9e9fb-3362-48cf-b2ec-3db89f981464/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cea9e9fb-3362-48cf-b2ec-3db89f981464/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d72732eb-0380-4f65-ab26-a788f64ad3fd/volumes/kubernetes.io~projected/kube-api-access-7mk5k:{mountpoint:/var/lib/kubelet/pods/d72732eb-0380-4f65-ab26-a788f64ad3fd/volumes/kubernetes.io~projected/kube-api-access-7mk5k major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff/volumes/kubernetes.io~projected/kube-api-access-czhmx:{mountpoint:/var/lib/kubelet/pods/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff/volumes/kubernetes.io~projected/kube-api-access-czhmx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e01f94e8-cf07-45d8-b023-0f87e7d41cdf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e01f94e8-cf07-45d8-b023-0f87e7d41cdf/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~projected/kube-api-access-gcswr:{mountpoint:/var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~projected/kube-api-access-gcswr major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~secret/stats-auth major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9337a97-e8fa-4061-ba5f-fea449c7d390/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e9337a97-e8fa-4061-ba5f-fea449c7d390/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1f800e1-24b3-4323-97f3-2009efb63d1b/volumes/kubernetes.io~projected/kube-api-access-bw8zr:{mountpoint:/var/lib/kubelet/pods/f1f800e1-24b3-4323-97f3-2009efb63d1b/volumes/kubernetes.io~projected/kube-api-access-bw8zr major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f237b168-098b-4df7-a56e-f245691a8086/volumes/kubernetes.io~projected/kube-api-access-8zj9w:{mountpoint:/var/lib/kubelet/pods/f237b168-098b-4df7-a56e-f245691a8086/volumes/kubernetes.io~projected/kube-api-access-8zj9w major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f70b67b3-d8eb-4065-8650-b26de9bc9f7c/volumes/kubernetes.io~projected/kube-api-access-f6bd5:{mountpoint:/var/lib/kubelet/pods/f70b67b3-d8eb-4065-8650-b26de9bc9f7c/volumes/kubernetes.io~projected/kube-api-access-f6bd5 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volumes/kubernetes.io~projected/kube-api-access-49vxv:{mountpoint:/var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volumes/kubernetes.io~projected/kube-api-access-49vxv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/df4c0da101fb391de7b13eeb540caaa95b1e879188c2869aea6e9fedc49264fb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ffe20b3ad2ad784583dad8e13525d914dc340fde0a9bd5ca6b01834067c67b8b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ebb90d5809cbc0a73f406db6d11217e80728aa43485d40e95fc26d7c25fb26b1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b9d95d59943bb77316cd725f1b0eb57d5677487074b2bfccb208af67f93d5209/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3143081d235cf68ac0b83d10696fc4d4d2bfd6675c3ad4ffd9bc0492987716cf/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/68e764a9b34278fbe947faca275f9ae71879a9fa9d292bc8648b3bba342599a4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/aedbd8a92d1bd3afb68ada6b2e4916c04d44e107a96873e3c66bc3720e1c9af1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f3e1d40754b156986aec15f6a11ef59a9b9585efa75d2d0f2f3ec3583da416b1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/5a10d259c9cf6cae8d1a1e8e6eb6ac67b54a6e70d9a3ac9ff2092464d383812b/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/3545bca697f7dabe5f3e90b38e29b4ddd66fcb179599492f95a9368d2e6be8a6/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f46b3e1f7afc97e1eb91c552382cb946564bf550be4e4a84d4b6fd2247c2fb04/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/382ebe337326eb24d1fa1fc21d65b2a8a8cd58bb4df8efa1d73c82b4835a84c3/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/10b917e86083af70b71080d683225ae4e55d1e07976326ba6273d159f234fa77/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/fc8953070767ed280ae31113ff2830d6e70304f8380998e59ca80b8ec05f475a/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9a9e2eee25568bc0b43fb7dbffaa5ad1096d9eeb8d51fceba978ed4f06575ee9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/63b078163d7714537559d39bea25f6c23bbc14d72194130fa0d85627d48f7133/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/54dd2762999d3264f2418ba1bd9493c64d59b0a5539b1a77802444ba06bee1b6/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/aaf1f04ee5137fbb697b898dac44ff25e4c4ad67e02fd56deedfcb36c8322c84/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/aa890b300e813dd8d14b6e7107c3f4c60d5b66b2a9837ce80e42c5d41dd1cbfb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/c6659ddf84e534f5a5762b0bf0828cf8c89607ac061e267ed3d83e7df8630d53/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/53f4a8333cc6acb507bb8a51caff1679f3e6ebbb9bd0ecdc3b74ac3311328bae/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6bbec2726d1773c5d239b1d5e8ad5b43bc79f4cd8dd0931512e918caaeca9f77/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7ceefb66c1d25c26dd961c8b0cf710b8e45664f4a1751bbede79f213442f5fe5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/cbee56b026c5f0ab4356c8049122b24a63632d8537d77bb2624565517caacf7b/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/fa2dd79643b6676ff1219a88471cfcc53270a23b0e16b96ee12a46553e13c7c4/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/397d6c15f295222a34bcbbd711bcce9781d452ba293095690cdb398106b933d5/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/351758d7c219994d65aa4c2e2e27a03476fa9355c2557100a28364ca4de27cd4/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e3175dd939091a2a5e53b6535dd7fcbbb9cd50dfd7d0f67f2ddc0a04e1fbc1f2/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/86df46785e9c1200cde9a83130339f038ade4d6eb568bf5e6a40ae6bc7ac975a/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/8aa874d85186e4ab8eb5eb5eaf9af4ee5211bfa697217f29c6858692beee0ef8/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/896a1af323accfac60c7b88fb5770c8a00293c0e47c7d9d819d11bb1a179e9f1/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/6ff55379fd712007b23484f065a5b4b51a5be8ca3d490cf578dc5899d4215752/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/829c6176c978ac8daafc492976d44f3949c48594841e5d40dafc9da0e16c65e9/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/7e5c4078f3ac13b71c50027d7e6a973365c7c00d52d5d7cb11cd7fbf819a9ed3/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/f1e602dedd4a7fc291e9733dc841157b4af3d08bc92d2ce93ad7daee62242921/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/b6e32c7c507c6a24ecbf063219649f8ceddf9ceacb06675f28daef8a151a10c3/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/d9cefa702cee35eec0cd4335b08c33c37b7fbe86357f2d3182364a93e772e6fe/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/714f694bc95f3d05adb8373c3bd04278e04a7589da0638db994977d11733d9b7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/9e1a6f329e53cd5e89fb4a2f8abd7e32b14c9748de2776c0e83c64fe3fc72cfd/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/19f4e54dd5959ff0586935cb4018139ecc4eeb237fcd3a6b87758bd96d0584b0/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/e2cd5f785ce5af4d400ad612718017eef38e9630d8f71677519ef56e9e21a05b/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/afa6476bb3e98c8f0652970548f74fbc39f333d56a7dc42cd600d489e0e41336/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/66b57cd04b3be724412d4e23d0d92c9bb94549a5573c895801c8bd1cdaf44387/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/f84b3b6ce3fdb16d86c7caa5633b8730783e2350cb00d4b3df5aaa1c43e3c3b3/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/97f8e5720b84f4fb7ffda3ed5a8e1e0fa3c3a641422ba5cd69e658397c1f6ed4/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/cd62167191b301d7e03bc3f438aa0dd67953ef1a7eced89a1671ce1524957a8a/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/48de49a0003041ac8ceac3b2e19f2ae046add2e26cc4c410390de4b02c7bd1bb/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ba0727526610d643b8cf602e262509593583fcadd268ee2a563d5d40ba8a0203/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8cc44349da8b1528bbc8dbd56f2fb095502e086f4cf53306004d227a04dad7a9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a40ed07e566b60815e4195a089ec404725c1e6a9b47b6c7091abc9da76624bd4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/141ed23901226b9584a977d51d35eebeadcbce6f06a870dd3be983da6b0925e8/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5d8c144fb76914381ed67f8a02d21dcc37021957a3709e959541bf5577d9e97c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b8ab5c33fd813096ecfafa8ea0327eda6a208a323c397a205376a3f316172054/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/b16159e98e496f8b13a8c72a477feaa926aa281d0edba508bde1ce91d1d3ab0f/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/aedae355566a5a73b1444c6bbc84eb715c6eb3079f5bc980ac0f7766e1e482d8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b9b17923e6196347c3085e13f2cf6aced9155fa474d6bd3cd804a72d8a6daef8/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914081 4911 manager.go:217] Machine: {Timestamp:2026-11-01 15:47:39.9136125 +0000 UTC m=+0.058067136 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099839 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fc9a0f9bec36d5882730fb29f82e SystemUUID:ec26fc9a-0f9b-ec36-d588-2730fb29f82e BootID:03a9c7f7-ff9e-4446-abd0-3f76f79c329c Filesystems:[{Device:/var/lib/kubelet/pods/1a3d24bb-ebe0-4623-88d7-77cd90a8b142/volumes/kubernetes.io~projected/kube-api-access-gbwmh DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04/volumes/kubernetes.io~projected/kube-api-access-vsgh4 DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volumes/kubernetes.io~projected/kube-api-access-49vxv DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6479c4becc1497c24f39a4338481f57c23b75a7cbdf2187c8035fd0128dc6a89/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f6d982a-e0d9-42ee-b342-290a02120b36/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a762abb23c5ff323d4eca7f380e7ed7ef7d9f1803db31837f1c7a0ef8edf8c33/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/57440ce3-722e-4835-927a-e8f2fac8b3c7/volumes/kubernetes.io~projected/kube-api-access-p48zp DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb341de1665cba78f97eccebff37648402746cddbd8e6bce65b57f33ed6ecdfa/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e085d33-5390-4d2e-a616-61160ee87e46/volumes/kubernetes.io~projected/kube-api-access-pw4mn DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebc811ca5f4ba37651300825ae8721966cfb27a227e21d6a99b0eb86b18d70a8/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5c3860132f0930f0440ed7fecbf576abe29003d8dc1f21d98aa86a0f3118838/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5671cc22-cecc-4d28-9054-9eee4e8dae58/volumes/kubernetes.io~projected/kube-api-access-pzh9b DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6433c5acfa2b7f0a4a44db14196a2443c020c4f96ace892d41563527ac62e3f1/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f70b67b3-d8eb-4065-8650-b26de9bc9f7c/volumes/kubernetes.io~projected/kube-api-access-f6bd5 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc84d1fcd7861299dd4b424fa3058fdd8bf0cbd0601ecda51f31e7a81fb84bc9/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9337a97-e8fa-4061-ba5f-fea449c7d390/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e98087dbe24fa28c1d6fe98985df40ea9e3014555c2642a2e85c2885b79ccde9/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a515047-c16a-4352-acaf-16b5bd0dea55/volumes/kubernetes.io~projected/kube-api-access-4qd88 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f56ebf77df15b3f21f331172314cd8afccf9a4ed636db7491ff0591399f71cc8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f6d982a-e0d9-42ee-b342-290a02120b36/volumes/kubernetes.io~projected/kube-api-access-598sq DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/883aac2f75972cbd239bde86626d7aa6d14a591b4033e01619f1ca690e03126b/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5718d617383e9595df537f321da8012d69f7641236d0056d0fcc7e84c3ffa8c3/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc9cc25584a67745436631926c54e00ee56c57b1a8816271364ef4df4e6a2ada/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f237b168-098b-4df7-a56e-f245691a8086/volumes/kubernetes.io~projected/kube-api-access-8zj9w DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cea9e9fb-3362-48cf-b2ec-3db89f981464/volumes/kubernetes.io~projected/kube-api-access-xmfn8 DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a515047-c16a-4352-acaf-16b5bd0dea55/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b220d197-999f-4b20-af10-f9ffb9420614/volumes/kubernetes.io~projected/kube-api-access-c9k9p DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a901c1f-0f50-4d5e-94e8-03326a1c650f/volumes/kubernetes.io~projected/kube-api-access-rt658 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a7cd5bd-620b-4d98-b64a-4582788161a4/volumes/kubernetes.io~projected/kube-api-access-rj4tt DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b24f7e8-b574-4609-96e5-9908302ac283/volumes/kubernetes.io~projected/kube-api-access-j892r DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36fdfbd3e39437309b5783cfe1676fb681d59ff75d539f3a74e983fd199dae7f/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29210824966f85f75f01690b769716d5e8887c2b8f87d4f2c4082b717e52b402/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f05b349f0fc079a118d6224026e3e1e2c0fdb353b5aa8c8f8668f2af89230e83/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e01f94e8-cf07-45d8-b023-0f87e7d41cdf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff/volumes/kubernetes.io~projected/kube-api-access-czhmx DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23a31617-3eb9-499e-a192-f4d24a34e835/volumes/kubernetes.io~projected/kube-api-access-gvm7b DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3be78730-9048-43e9-9d31-e3819027d095/volumes/kubernetes.io~projected/kube-api-access-kqng2 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9578d6292af38dda6b140d751816070e36652fe4e62e1cd9d0c2581f1c2ce3d2/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cea9e9fb-3362-48cf-b2ec-3db89f981464/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a9662ce2-6f58-4385-8f20-f9ed018fea61/volumes/kubernetes.io~projected/kube-api-access-8vfx8 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b5fc71dfd363f6bafc11949156b7d6c211fdcd3e48869d87052b244572aee24/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes/kubernetes.io~projected/kube-api-access-4m465 DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~projected/kube-api-access-6rj4g DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09cf87aa-e08e-4890-acf5-c8da31cd08da/volumes/kubernetes.io~projected/kube-api-access-t4vwg DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7423d5afff283d04bc8ad7e0c1eace43c70347870211451fc6cd8d48f6993863/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23a31617-3eb9-499e-a192-f4d24a34e835/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9662ce2-6f58-4385-8f20-f9ed018fea61/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f440e8a-b448-4d18-9306-c5627dd6f5ad/volumes/kubernetes.io~projected/kube-api-access-227hr DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd9c772427de48f7c44ad67b56abd3ba59a0c928a6379a2c6772350b6e91c667/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85d4d845-68cc-44e8-9503-5969ed5fcc1b/volumes/kubernetes.io~projected/kube-api-access-zldzc DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d72732eb-0380-4f65-ab26-a788f64ad3fd/volumes/kubernetes.io~projected/kube-api-access-7mk5k DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7039a946-2ec8-414d-9093-7f355172fba4/volumes/kubernetes.io~projected/kube-api-access-x9vgf DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/volumes/kubernetes.io~projected/kube-api-access-gcswr DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/869c87fc97a4e70718c6ce199b5f1dce0857043325eeacf13421ce2ff0914464/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff05d7d5-9766-49fb-86de-0c02df3dc0af/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1f800e1-24b3-4323-97f3-2009efb63d1b/volumes/kubernetes.io~projected/kube-api-access-bw8zr DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a56f9cfa54ee501e104549baa9964260c9ce82087d7d9197157ffc5284c84180/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:29210824966f85f MacAddress:d2:66:45:22:05:51 Speed:10000 Mtu:8901} {Name:36fdfbd3e394373 MacAddress:06:32:d6:bf:3c:1b Speed:10000 Mtu:8901} {Name:7423d5afff283d0 MacAddress:be:bb:a4:0e:74:09 Speed:10000 Mtu:8901} {Name:869c87fc97a4e70 MacAddress:0e:04:46:b0:f2:07 Speed:10000 Mtu:8901} {Name:883aac2f75972cb MacAddress:c6:12:0e:55:3a:62 Speed:10000 Mtu:8901} {Name:9578d6292af38dd MacAddress:ea:8f:72:07:6b:56 Speed:10000 Mtu:8901} {Name:9b5fc71dfd363f6 MacAddress:36:f0:ab:79:ad:e1 Speed:10000 Mtu:8901} {Name:a56f9cfa54ee501 MacAddress:be:66:83:78:29:a6 Speed:10000 Mtu:8901} {Name:a5c3860132f0930 MacAddress:5a:3a:b5:07:2f:0a Speed:10000 Mtu:8901} {Name:a762abb23c5ff32 MacAddress:fe:1d:d9:96:f9:ff Speed:10000 Mtu:8901} {Name:bc9cc25584a6774 MacAddress:42:d9:0c:14:23:8e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e4:35:ee:a1 Speed:0 Mtu:9001} {Name:br-int MacAddress:92:ee:9b:63:15:f7 Speed:0 Mtu:8901} {Name:ebc811ca5f4ba37 MacAddress:7a:bf:ad:2b:07:15 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e4:35:ee:a1 Speed:0 Mtu:9001} {Name:f05b349f0fc079a MacAddress:c2:d5:3f:9b:c5:28 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c6:15:28:ff:57:6d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:98:e7:b0:88:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914428 4911 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914475 4911 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914757 4911 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914778 4911 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-101.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914895 4911 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914902 4911 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 15:47:39.914921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914912 4911 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914933 4911 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.914952 4911 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915110 4911 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915215 4911 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915247 4911 kubelet.go:493] "Attempting to sync node with API server" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915256 4911 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915269 4911 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915278 4911 kubelet.go:405] "Adding apiserver pod source" Nov 01 15:47:39.915342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915286 4911 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 15:47:39.915983 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915970 4911 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 15:47:39.915983 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.915983 4911 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:39.916504 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.916479 4911 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 15:47:39.918598 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918584 4911 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 15:47:39.918598 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918598 4911 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:39.918697 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918688 4911 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 15:47:39.918868 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918858 4911 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 15:47:39.918898 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918887 4911 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 15:47:39.918982 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918973 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 15:47:39.919012 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918987 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 15:47:39.919012 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.918994 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 15:47:39.919012 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919000 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 15:47:39.919012 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919006 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 15:47:39.919012 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919011 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919019 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919028 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919052 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919059 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919076 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919085 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919113 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 15:47:39.919165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.919121 4911 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 15:47:39.922851 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.922838 4911 server.go:1263] "Started kubelet" Nov 01 15:47:39.922937 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.922911 4911 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 15:47:39.923381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.923142 4911 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 15:47:39.923381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.923216 4911 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 15:47:39.923427 ip-10-0-143-101 systemd[1]: Started Kubernetes Kubelet. Nov 01 15:47:39.925706 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.925682 4911 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 15:47:39.928017 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.927999 4911 server.go:317] "Adding debug handlers to kubelet server" Nov 01 15:47:39.930979 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.930956 4911 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:39.931967 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.931950 4911 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 15:47:39.932072 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.931963 4911 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:39.932072 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.932024 4911 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 15:47:39.932072 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.932052 4911 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 15:47:39.932223 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.932106 4911 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:00 +0000 UTC" deadline="2028-06-27 16:18:30.947529033 +0000 UTC" Nov 01 15:47:39.932223 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.932147 4911 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14496h30m51.015385871s" Nov 01 15:47:39.932223 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.932170 4911 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 15:47:39.933661 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.933640 4911 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:39.934708 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934680 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7cd5bd-620b-4d98-b64a-4582788161a4" volumeName="kubernetes.io/projected/1a7cd5bd-620b-4d98-b64a-4582788161a4-kube-api-access-rj4tt" seLinuxMountContext="" Nov 01 15:47:39.934708 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934707 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a31617-3eb9-499e-a192-f4d24a34e835" volumeName="kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-snapshots" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934716 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a31617-3eb9-499e-a192-f4d24a34e835" volumeName="kubernetes.io/secret/23a31617-3eb9-499e-a192-f4d24a34e835-serving-cert" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934724 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab1985-6883-4db6-ac14-0d3ae3159c20" volumeName="kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934732 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5432881f-c368-4d90-b8d2-4958ba502338" volumeName="kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934738 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57440ce3-722e-4835-927a-e8f2fac8b3c7" volumeName="kubernetes.io/configmap/57440ce3-722e-4835-927a-e8f2fac8b3c7-telemetry-config" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934746 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a31617-3eb9-499e-a192-f4d24a34e835" volumeName="kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-service-ca-bundle" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934753 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f440e8a-b448-4d18-9306-c5627dd6f5ad" volumeName="kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-tmp" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934764 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5432881f-c368-4d90-b8d2-4958ba502338" volumeName="kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934771 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5671cc22-cecc-4d28-9054-9eee4e8dae58" volumeName="kubernetes.io/projected/5671cc22-cecc-4d28-9054-9eee4e8dae58-kube-api-access-pzh9b" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934783 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b24f7e8-b574-4609-96e5-9908302ac283" volumeName="kubernetes.io/projected/8b24f7e8-b574-4609-96e5-9908302ac283-kube-api-access-j892r" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934791 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d72732eb-0380-4f65-ab26-a788f64ad3fd" volumeName="kubernetes.io/projected/d72732eb-0380-4f65-ab26-a788f64ad3fd-kube-api-access-7mk5k" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934798 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff" volumeName="kubernetes.io/configmap/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-iptables-alerter-script" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934805 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e01f94e8-cf07-45d8-b023-0f87e7d41cdf" volumeName="kubernetes.io/secret/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-agent-certs" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934814 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e085d33-5390-4d2e-a616-61160ee87e46" volumeName="kubernetes.io/configmap/0e085d33-5390-4d2e-a616-61160ee87e46-config-volume" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934821 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a31617-3eb9-499e-a192-f4d24a34e835" volumeName="kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-trusted-ca-bundle" seLinuxMountContext="" Nov 01 15:47:39.934823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934829 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a31617-3eb9-499e-a192-f4d24a34e835" volumeName="kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-tmp" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934837 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab1985-6883-4db6-ac14-0d3ae3159c20" volumeName="kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934844 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5432881f-c368-4d90-b8d2-4958ba502338" volumeName="kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934851 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b220d197-999f-4b20-af10-f9ffb9420614" volumeName="kubernetes.io/empty-dir/b220d197-999f-4b20-af10-f9ffb9420614-data-volume" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934858 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff05d7d5-9766-49fb-86de-0c02df3dc0af" volumeName="kubernetes.io/secret/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934865 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f440e8a-b448-4d18-9306-c5627dd6f5ad" volumeName="kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-tuned" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934872 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3be78730-9048-43e9-9d31-e3819027d095" volumeName="kubernetes.io/projected/3be78730-9048-43e9-9d31-e3819027d095-kube-api-access-kqng2" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934879 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a901c1f-0f50-4d5e-94e8-03326a1c650f" volumeName="kubernetes.io/projected/7a901c1f-0f50-4d5e-94e8-03326a1c650f-kube-api-access-rt658" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934886 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6d982a-e0d9-42ee-b342-290a02120b36" volumeName="kubernetes.io/secret/8f6d982a-e0d9-42ee-b342-290a02120b36-cert" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934892 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d66b38-3dfa-46eb-a2c7-d8a5de28be89" volumeName="kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-stats-auth" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934899 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f237b168-098b-4df7-a56e-f245691a8086" volumeName="kubernetes.io/configmap/f237b168-098b-4df7-a56e-f245691a8086-serviceca" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934905 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff05d7d5-9766-49fb-86de-0c02df3dc0af" volumeName="kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-config" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934915 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f440e8a-b448-4d18-9306-c5627dd6f5ad" volumeName="kubernetes.io/projected/2f440e8a-b448-4d18-9306-c5627dd6f5ad-kube-api-access-227hr" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934923 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5432881f-c368-4d90-b8d2-4958ba502338" volumeName="kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934931 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9662ce2-6f58-4385-8f20-f9ed018fea61" volumeName="kubernetes.io/secret/a9662ce2-6f58-4385-8f20-f9ed018fea61-serving-cert" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934947 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff" volumeName="kubernetes.io/projected/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-kube-api-access-czhmx" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934954 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a31617-3eb9-499e-a192-f4d24a34e835" volumeName="kubernetes.io/projected/23a31617-3eb9-499e-a192-f4d24a34e835-kube-api-access-gvm7b" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934962 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e085d33-5390-4d2e-a616-61160ee87e46" volumeName="kubernetes.io/empty-dir/0e085d33-5390-4d2e-a616-61160ee87e46-tmp-dir" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934972 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85d4d845-68cc-44e8-9503-5969ed5fcc1b" volumeName="kubernetes.io/projected/85d4d845-68cc-44e8-9503-5969ed5fcc1b-kube-api-access-zldzc" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934979 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1f800e1-24b3-4323-97f3-2009efb63d1b" volumeName="kubernetes.io/projected/f1f800e1-24b3-4323-97f3-2009efb63d1b-kube-api-access-bw8zr" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934987 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab1985-6883-4db6-ac14-0d3ae3159c20" volumeName="kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.934993 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04" volumeName="kubernetes.io/projected/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04-kube-api-access-vsgh4" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935000 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5432881f-c368-4d90-b8d2-4958ba502338" volumeName="kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935009 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7039a946-2ec8-414d-9093-7f355172fba4" volumeName="kubernetes.io/projected/7039a946-2ec8-414d-9093-7f355172fba4-kube-api-access-x9vgf" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935016 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9662ce2-6f58-4385-8f20-f9ed018fea61" volumeName="kubernetes.io/configmap/a9662ce2-6f58-4385-8f20-f9ed018fea61-config" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935024 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea9e9fb-3362-48cf-b2ec-3db89f981464" volumeName="kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-config" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935031 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea9e9fb-3362-48cf-b2ec-3db89f981464" volumeName="kubernetes.io/projected/cea9e9fb-3362-48cf-b2ec-3db89f981464-kube-api-access-xmfn8" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935055 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff05d7d5-9766-49fb-86de-0c02df3dc0af" volumeName="kubernetes.io/projected/ff05d7d5-9766-49fb-86de-0c02df3dc0af-kube-api-access-49vxv" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935067 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6d982a-e0d9-42ee-b342-290a02120b36" volumeName="kubernetes.io/projected/8f6d982a-e0d9-42ee-b342-290a02120b36-kube-api-access-598sq" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935079 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a901c1f-0f50-4d5e-94e8-03326a1c650f" volumeName="kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cni-binary-copy" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935092 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a515047-c16a-4352-acaf-16b5bd0dea55" volumeName="kubernetes.io/configmap/8a515047-c16a-4352-acaf-16b5bd0dea55-config" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935103 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea9e9fb-3362-48cf-b2ec-3db89f981464" volumeName="kubernetes.io/secret/cea9e9fb-3362-48cf-b2ec-3db89f981464-serving-cert" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935115 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9337a97-e8fa-4061-ba5f-fea449c7d390" volumeName="kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935126 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab1985-6883-4db6-ac14-0d3ae3159c20" volumeName="kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935139 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57440ce3-722e-4835-927a-e8f2fac8b3c7" volumeName="kubernetes.io/projected/57440ce3-722e-4835-927a-e8f2fac8b3c7-kube-api-access-p48zp" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935150 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9662ce2-6f58-4385-8f20-f9ed018fea61" volumeName="kubernetes.io/projected/a9662ce2-6f58-4385-8f20-f9ed018fea61-kube-api-access-8vfx8" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935162 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f237b168-098b-4df7-a56e-f245691a8086" volumeName="kubernetes.io/projected/f237b168-098b-4df7-a56e-f245691a8086-kube-api-access-8zj9w" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935173 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f70b67b3-d8eb-4065-8650-b26de9bc9f7c" volumeName="kubernetes.io/projected/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-kube-api-access-f6bd5" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935184 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f38b5c-f14d-48fd-b0cf-a2581c62384a" volumeName="kubernetes.io/configmap/19f38b5c-f14d-48fd-b0cf-a2581c62384a-nginx-conf" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935195 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b220d197-999f-4b20-af10-f9ffb9420614" volumeName="kubernetes.io/projected/b220d197-999f-4b20-af10-f9ffb9420614-kube-api-access-c9k9p" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935214 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d72732eb-0380-4f65-ab26-a788f64ad3fd" volumeName="kubernetes.io/empty-dir/d72732eb-0380-4f65-ab26-a788f64ad3fd-tmp-dir" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935225 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff05d7d5-9766-49fb-86de-0c02df3dc0af" volumeName="kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-script-lib" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935237 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" volumeName="kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935248 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" volumeName="kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-binary-copy" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935259 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5432881f-c368-4d90-b8d2-4958ba502338" volumeName="kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935271 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5432881f-c368-4d90-b8d2-4958ba502338" volumeName="kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets" seLinuxMountContext="" Nov 01 15:47:39.935269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935283 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d66b38-3dfa-46eb-a2c7-d8a5de28be89" volumeName="kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-default-certificate" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935294 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f70b67b3-d8eb-4065-8650-b26de9bc9f7c" volumeName="kubernetes.io/configmap/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-config" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935307 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09cf87aa-e08e-4890-acf5-c8da31cd08da" volumeName="kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935318 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab1985-6883-4db6-ac14-0d3ae3159c20" volumeName="kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935331 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab1985-6883-4db6-ac14-0d3ae3159c20" volumeName="kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935343 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a515047-c16a-4352-acaf-16b5bd0dea55" volumeName="kubernetes.io/secret/8a515047-c16a-4352-acaf-16b5bd0dea55-serving-cert" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935354 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e01f94e8-cf07-45d8-b023-0f87e7d41cdf" volumeName="kubernetes.io/configmap/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-konnectivity-ca" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935365 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a515047-c16a-4352-acaf-16b5bd0dea55" volumeName="kubernetes.io/projected/8a515047-c16a-4352-acaf-16b5bd0dea55-kube-api-access-4qd88" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935377 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cea9e9fb-3362-48cf-b2ec-3db89f981464" volumeName="kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-trusted-ca" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935388 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff05d7d5-9766-49fb-86de-0c02df3dc0af" volumeName="kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-env-overrides" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935399 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" volumeName="kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935410 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eab1985-6883-4db6-ac14-0d3ae3159c20" volumeName="kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935421 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a901c1f-0f50-4d5e-94e8-03326a1c650f" volumeName="kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-daemon-config" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935432 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3d66b38-3dfa-46eb-a2c7-d8a5de28be89" volumeName="kubernetes.io/projected/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-kube-api-access-gcswr" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935444 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e085d33-5390-4d2e-a616-61160ee87e46" volumeName="kubernetes.io/projected/0e085d33-5390-4d2e-a616-61160ee87e46-kube-api-access-pw4mn" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935455 4911 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" volumeName="kubernetes.io/projected/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-kube-api-access-gbwmh" seLinuxMountContext="" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935475 4911 reconstruct.go:97] "Volume reconstruction finished" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935482 4911 reconciler.go:29] "Reconciler: start to sync state" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.935525 4911 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.936000 4911 factory.go:55] Registering systemd factory Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.936017 4911 factory.go:223] Registration of the systemd container factory successfully Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.936468 4911 factory.go:153] Registering CRI-O factory Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.936480 4911 factory.go:223] Registration of the crio container factory successfully Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.936530 4911 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.936550 4911 factory.go:103] Registering Raw factory Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.936565 4911 manager.go:1199] Started watching for new ooms in manager Nov 01 15:47:39.937171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.937076 4911 manager.go:319] Starting recovery of all containers Nov 01 15:47:39.938602 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.938578 4911 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 15:47:39.950664 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.950639 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/1.log" Nov 01 15:47:39.951810 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.951788 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/1.log" Nov 01 15:47:39.956312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.956286 4911 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 15:47:39.957893 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.957870 4911 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 15:47:39.957893 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.957895 4911 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 15:47:39.958117 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.957913 4911 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 15:47:39.958117 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:39.957952 4911 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 15:47:39.962779 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.962191 4911 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:39.984168 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.984126 4911 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="90b75d7debc2809b64bb083b76a9b966a0984bceae31ed4c250cbcb686170d70" exitCode=0 Nov 01 15:47:39.984168 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.984164 4911 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="3b8b7cf252a02e62760dbf5bc9ac9bbc47b0130ff2441a963a12bfa357390605" exitCode=0 Nov 01 15:47:39.984168 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.984170 4911 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="64d591b8b791dc0c6f1e4cfed2cf938b4c4853b65a464c927f9053ff21a7dabf" exitCode=0 Nov 01 15:47:39.984168 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.984176 4911 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="d2af07a4d9e76c3c1c27c9ee0986457a1a81d6dc8828bfd747dff434b619f7d6" exitCode=0 Nov 01 15:47:39.984391 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.984182 4911 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="c4544ee5d4cae955421459b530c9a54f692fb94f6f931e92e7a4639731dc2254" exitCode=0 Nov 01 15:47:39.984391 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.984189 4911 generic.go:356] "Generic (PLEG): container finished" podID="1a3d24bb-ebe0-4623-88d7-77cd90a8b142" containerID="58ed2727a8111e45c58ba543d2a3a2c56789ba42e698a8726fb837c84cb2f122" exitCode=0 Nov 01 15:47:39.987102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.987080 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/1.log" Nov 01 15:47:39.987207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:39.987107 4911 generic.go:356] "Generic (PLEG): container finished" podID="cea9e9fb-3362-48cf-b2ec-3db89f981464" containerID="f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66" exitCode=255 Nov 01 15:47:40.001605 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.001574 4911 generic.go:356] "Generic (PLEG): container finished" podID="ae69c9ba8347dbb2dbf47900d239e09c" containerID="4a78469e53da3ad7ade2ed9c8b5883c4ec955caae55cf437bd4678f410beb5c4" exitCode=0 Nov 01 15:47:40.025814 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.025795 4911 manager.go:324] Recovery completed Nov 01 15:47:40.044674 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044649 4911 cpu_manager.go:226] "Starting" policy="none" Nov 01 15:47:40.044674 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044667 4911 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 15:47:40.044806 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044687 4911 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 15:47:40.044840 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044812 4911 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 15:47:40.044840 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044822 4911 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 15:47:40.044840 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044838 4911 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 15:47:40.044933 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044843 4911 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 15:47:40.044933 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044850 4911 policy_none.go:50] "Start" Nov 01 15:47:40.044933 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044860 4911 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 15:47:40.044933 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.044872 4911 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 15:47:40.045062 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.045001 4911 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 15:47:40.045062 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.045008 4911 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 15:47:40.045062 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.045014 4911 policy_none.go:44] "Start" Nov 01 15:47:40.050490 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.050462 4911 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 15:47:40.053218 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.053180 4911 manager.go:341] "Starting Device Plugin manager" Nov 01 15:47:40.053218 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:40.053205 4911 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 15:47:40.053218 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.053215 4911 server.go:87] "Starting device plugin registration server" Nov 01 15:47:40.053961 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.053946 4911 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 15:47:40.053999 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.053962 4911 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 15:47:40.054093 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.054073 4911 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 15:47:40.054217 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.054185 4911 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 15:47:40.054276 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.054227 4911 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 15:47:40.054276 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.054234 4911 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 15:47:40.054374 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.054285 4911 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T15:47:40.054216266Z","UUID":"6829c2b4-0f63-4376-af59-dd452898b039","Handler":null,"Name":"","Endpoint":""} Nov 01 15:47:40.055479 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.055455 4911 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 15:47:40.055561 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.055485 4911 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 15:47:40.055851 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:40.055789 4911 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 15:47:40.058030 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.057986 4911 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal"] Nov 01 15:47:40.058738 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.058720 4911 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.058820 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.058710 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"fd6f673b54e13544e2f8008a04d167a7","Type":"ContainerStarted","Data":"598fcd24bfd609286f254f940026addee5e6e751695dc03ef8e5553aff107693"} Nov 01 15:47:40.058888 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.058840 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"fd6f673b54e13544e2f8008a04d167a7","Type":"ContainerStarted","Data":"0b8c8b3c8f4500e8480c472eb22fe8ca6a53c7a7a55de75763158f3963eebb7d"} Nov 01 15:47:40.058937 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.058901 4911 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.058983 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.058944 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"ae69c9ba8347dbb2dbf47900d239e09c","Type":"ContainerStarted","Data":"d0cd20ea7e1c5a3f4a403907af8d7786d5055fcc8f1ff6895c2ea212b050f5f4"} Nov 01 15:47:40.058983 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.058968 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"ae69c9ba8347dbb2dbf47900d239e09c","Type":"ContainerDied","Data":"4a78469e53da3ad7ade2ed9c8b5883c4ec955caae55cf437bd4678f410beb5c4"} Nov 01 15:47:40.059104 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.058988 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" event={"ID":"ae69c9ba8347dbb2dbf47900d239e09c","Type":"ContainerStarted","Data":"241963f3617be9c09b00d3189ac0b3bf65a23ab3b78ef2f04c7479df01411b65"} Nov 01 15:47:40.071223 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.071201 4911 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:40.071318 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:40.071248 4911 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.071454 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.071441 4911 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:40.071511 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:40.071487 4911 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.139285 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.139254 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.139285 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.139287 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.139433 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.139305 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd6f673b54e13544e2f8008a04d167a7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"fd6f673b54e13544e2f8008a04d167a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.154376 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.154350 4911 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:40.155712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.155693 4911 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:40.155796 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.155725 4911 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:40.155796 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.155735 4911 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-101.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:40.155796 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.155779 4911 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.168090 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.168072 4911 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.168139 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.168121 4911 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.240191 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.240161 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.240304 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.240198 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.240304 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.240264 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.240383 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.240333 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae69c9ba8347dbb2dbf47900d239e09c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"ae69c9ba8347dbb2dbf47900d239e09c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.240414 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.240378 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd6f673b54e13544e2f8008a04d167a7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"fd6f673b54e13544e2f8008a04d167a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.240414 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.240409 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd6f673b54e13544e2f8008a04d167a7-config\") pod \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" (UID: \"fd6f673b54e13544e2f8008a04d167a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:40.916276 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.916237 4911 apiserver.go:52] "Watching apiserver" Nov 01 15:47:40.931879 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.931853 4911 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 15:47:40.932530 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.932501 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j46fh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv","openshift-console-operator/console-operator-7cd56b8dbc-5qhqp","kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal","openshift-dns/dns-default-jx495","kube-system/konnectivity-agent-wvndh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99","openshift-network-operator/iptables-alerter-9x9k2","openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx","openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7","openshift-cluster-node-tuning-operator/tuned-v77kh","openshift-dns/node-resolver-wx7cw","openshift-ingress-canary/ingress-canary-cf2mk","openshift-image-registry/image-registry-84cf9dbdbb-c9jvz","openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj","openshift-image-registry/image-registry-6fb79fc6b7-gq4fr","openshift-multus/multus-additional-cni-plugins-m8qrf","openshift-multus/multus-zn8p6","openshift-multus/network-metrics-daemon-l6qq5","kube-system/kas-connection-checker-685895c7c9-5qrs6","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7","openshift-image-registry/node-ca-czbft","openshift-insights/insights-runtime-extractor-6x8kz","openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9","kube-system/kas-connection-checker-685895c7c9-g4sxx","kube-system/global-pull-secret-syncer-xvq6f","openshift-ingress/router-default-69f6694997-vvcm2","openshift-ovn-kubernetes/ovnkube-node-hvbbz","openshift-insights/insights-operator-7c4986d449-fz7xr","openshift-network-diagnostics/network-check-target-bdfqx","kube-system/kas-connection-checker-685895c7c9-6v9fs"] Nov 01 15:47:40.933794 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.933773 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:40.934143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.934122 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:40.935963 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.935938 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:40.936418 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.936399 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:40.937341 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.937320 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.937341 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.937338 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:40.937472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.937409 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.937687 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.937668 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.938122 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.938101 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 15:47:40.938216 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.938200 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.938255 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.938228 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 15:47:40.938587 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.938571 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 15:47:40.939740 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.939425 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 15:47:40.939740 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.939508 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kkxnv\"" Nov 01 15:47:40.939740 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.939531 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 15:47:40.939923 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.939811 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2mxnc\"" Nov 01 15:47:40.941109 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.940423 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 15:47:40.941109 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.940508 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 15:47:40.941109 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.940714 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 15:47:40.941109 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.940891 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2665t\"" Nov 01 15:47:40.941958 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.941380 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:40.941958 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.941718 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 15:47:40.941958 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.941823 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 15:47:40.942143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.941976 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 15:47:40.943821 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.942608 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.943821 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.942774 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.943821 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.943206 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.943821 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.943252 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.943821 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.943586 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 15:47:40.943821 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.943692 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.944108 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.943866 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.944108 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.943878 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 15:47:40.944108 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.943899 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.944108 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.944012 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 15:47:40.944282 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.944248 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.944282 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.944274 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.945312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945093 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.945312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945141 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 15:47:40.945312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945155 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 15:47:40.945312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945202 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.945312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945243 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.945312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945267 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.945312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945294 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8bq96\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945334 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945340 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945373 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945391 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945298 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-slghr\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945430 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945538 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945563 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 15:47:40.945663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945618 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6498b\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945692 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945716 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945737 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945793 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945820 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945825 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8ck6h\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945860 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-28hsn\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945898 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5vhjr\"" Nov 01 15:47:40.946105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.945967 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 15:47:40.946390 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.946182 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 15:47:40.946468 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.946454 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 15:47:40.949149 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.949131 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 15:47:40.949597 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.949579 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:40.950083 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.950069 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jx495" Nov 01 15:47:40.951571 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.951544 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.951571 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.951568 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.951728 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.951708 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:40.952603 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.952570 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4p8cp\"" Nov 01 15:47:40.952687 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.952665 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 15:47:40.952760 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.952690 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 15:47:40.952760 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.952702 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 15:47:40.952884 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.952866 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 15:47:40.953028 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.953010 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 15:47:40.953201 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.953184 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 15:47:40.953201 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.953194 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 15:47:40.953201 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.953186 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.954306 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.954265 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jsz89\"" Nov 01 15:47:40.954392 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.954371 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.954452 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.954441 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.954652 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.954632 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 15:47:40.954992 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.954974 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58l9j\"" Nov 01 15:47:40.955088 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955011 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.955088 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955024 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.955209 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955191 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2bqxm\"" Nov 01 15:47:40.955339 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955325 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwtj5\"" Nov 01 15:47:40.955651 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955630 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qm9xj\"" Nov 01 15:47:40.955724 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955706 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 15:47:40.955830 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955813 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjc9m\"" Nov 01 15:47:40.955915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.955895 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 15:47:40.956077 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956059 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.956143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956097 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 15:47:40.956143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956124 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.956239 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956199 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x5r5\"" Nov 01 15:47:40.956389 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956371 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 15:47:40.956456 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956390 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 15:47:40.956505 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956474 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.956505 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956380 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 15:47:40.956624 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956609 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-srrqq\"" Nov 01 15:47:40.956780 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956759 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 15:47:40.956845 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956831 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 15:47:40.956901 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956846 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:40.956901 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956862 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtfck\"" Nov 01 15:47:40.956994 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956474 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 15:47:40.956994 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.956612 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 15:47:40.957102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957027 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wl7pm\"" Nov 01 15:47:40.957102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957033 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hxd8\"" Nov 01 15:47:40.957202 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957113 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ttxgn\"" Nov 01 15:47:40.957202 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957140 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.957286 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957202 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4c2bg\"" Nov 01 15:47:40.957286 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957238 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 15:47:40.957391 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957366 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 15:47:40.957491 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:40.957405 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:41.011269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.011241 4911 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:41.011431 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.011339 4911 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:41.022930 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.022909 4911 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:41.023002 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.022948 4911 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:41.023431 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.023407 4911 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:41.023542 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.023446 4911 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-101.us-east-2.compute.internal" Nov 01 15:47:41.034265 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.034245 4911 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 15:47:41.045694 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045668 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vgf\" (UniqueName: \"kubernetes.io/projected/7039a946-2ec8-414d-9093-7f355172fba4-kube-api-access-x9vgf\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.045765 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045699 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f237b168-098b-4df7-a56e-f245691a8086-host\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:41.045765 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045716 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6d982a-e0d9-42ee-b342-290a02120b36-cert\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:41.045765 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045732 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-env-overrides\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.045871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045782 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.045871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045812 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rj4g\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.045871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045832 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.045871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045849 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.046022 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045874 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-netns\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.046022 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045931 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-env-overrides\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.046022 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045975 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f6d982a-e0d9-42ee-b342-290a02120b36-cert\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:41.046022 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.045985 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.046173 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046018 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vxv\" (UniqueName: \"kubernetes.io/projected/ff05d7d5-9766-49fb-86de-0c02df3dc0af-kube-api-access-49vxv\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.046173 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046116 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.046237 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046225 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.046303 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046285 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57440ce3-722e-4835-927a-e8f2fac8b3c7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:41.046334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046313 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-config\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.046334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046329 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4mn\" (UniqueName: \"kubernetes.io/projected/0e085d33-5390-4d2e-a616-61160ee87e46-kube-api-access-pw4mn\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.046408 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046350 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-socket-dir-parent\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.046408 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046374 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-kubelet\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.046507 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046409 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.046507 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046437 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337a97-e8fa-4061-ba5f-fea449c7d390-original-pull-secret\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.046507 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046472 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8zr\" (UniqueName: \"kubernetes.io/projected/f1f800e1-24b3-4323-97f3-2009efb63d1b-kube-api-access-bw8zr\") pod \"volume-data-source-validator-58c9f5f46d-xs4tv\" (UID: \"f1f800e1-24b3-4323-97f3-2009efb63d1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" Nov 01 15:47:41.046635 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046508 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-bin\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.046635 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046535 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcswr\" (UniqueName: \"kubernetes.io/projected/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-kube-api-access-gcswr\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.046635 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046540 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-config\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.046635 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046582 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.046635 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046611 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57440ce3-722e-4835-927a-e8f2fac8b3c7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046642 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046663 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046772 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9662ce2-6f58-4385-8f20-f9ed018fea61-config\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046802 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mk5k\" (UniqueName: \"kubernetes.io/projected/d72732eb-0380-4f65-ab26-a788f64ad3fd-kube-api-access-7mk5k\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046824 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046845 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046862 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m465\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046884 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.046891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046888 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046909 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhmx\" (UniqueName: \"kubernetes.io/projected/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-kube-api-access-czhmx\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046934 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzh9b\" (UniqueName: \"kubernetes.io/projected/5671cc22-cecc-4d28-9054-9eee4e8dae58-kube-api-access-pzh9b\") pod \"migrator-66df5d496b-mz2hx\" (UID: \"5671cc22-cecc-4d28-9054-9eee4e8dae58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046962 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysconfig\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.046988 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047000 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9662ce2-6f58-4385-8f20-f9ed018fea61-config\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047015 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4tt\" (UniqueName: \"kubernetes.io/projected/1a7cd5bd-620b-4d98-b64a-4582788161a4-kube-api-access-rj4tt\") pod \"kas-connection-checker-685895c7c9-6v9fs\" (UID: \"1a7cd5bd-620b-4d98-b64a-4582788161a4\") " pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047061 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-modprobe-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047073 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047102 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-var-lib-kubelet\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047115 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047205 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-registration-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.047324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047309 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047348 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047371 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047405 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047434 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-iptables-alerter-script\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047526 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047554 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047588 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-log-socket\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047622 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047628 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-script-lib\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047658 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-netns\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047679 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-tmp\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047716 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047754 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-tmp\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047761 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.047791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047784 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-config\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047813 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047852 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047855 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-iptables-alerter-script\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047883 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-script-lib\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047908 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-host\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047936 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a515047-c16a-4352-acaf-16b5bd0dea55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047951 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-host-slash\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047953 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovnkube-config\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.047989 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-systemd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048078 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-etc-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048105 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048126 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048142 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19f38b5c-f14d-48fd-b0cf-a2581c62384a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048158 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-lib-modules\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048243 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e085d33-5390-4d2e-a616-61160ee87e46-config-volume\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048260 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-k8s-cni-cncf-io\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048303 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19f38b5c-f14d-48fd-b0cf-a2581c62384a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048332 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k9p\" (UniqueName: \"kubernetes.io/projected/b220d197-999f-4b20-af10-f9ffb9420614-kube-api-access-c9k9p\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048316 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a515047-c16a-4352-acaf-16b5bd0dea55-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048352 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048384 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-socket-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048439 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-snapshots\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048472 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23a31617-3eb9-499e-a192-f4d24a34e835-serving-cert\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.048471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048477 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e085d33-5390-4d2e-a616-61160ee87e46-config-volume\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048498 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm7b\" (UniqueName: \"kubernetes.io/projected/23a31617-3eb9-499e-a192-f4d24a34e835-kube-api-access-gvm7b\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048523 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-sys\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048550 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048561 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/23a31617-3eb9-499e-a192-f4d24a34e835-snapshots\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048574 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e085d33-5390-4d2e-a616-61160ee87e46-tmp-dir\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048604 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048616 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/23a31617-3eb9-499e-a192-f4d24a34e835-serving-cert\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048634 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048663 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e085d33-5390-4d2e-a616-61160ee87e46-tmp-dir\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048759 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227hr\" (UniqueName: \"kubernetes.io/projected/2f440e8a-b448-4d18-9306-c5627dd6f5ad-kube-api-access-227hr\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048839 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cea9e9fb-3362-48cf-b2ec-3db89f981464-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048856 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048891 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-systemd-units\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048908 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-netd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048954 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048975 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f237b168-098b-4df7-a56e-f245691a8086-serviceca\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048986 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.048995 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-system-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049012 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-daemon-config\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049028 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049063 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48zp\" (UniqueName: \"kubernetes.io/projected/57440ce3-722e-4835-927a-e8f2fac8b3c7-kube-api-access-p48zp\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049183 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f237b168-098b-4df7-a56e-f245691a8086-serviceca\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049184 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-var-lib-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049200 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049230 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-multus-certs\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049249 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt658\" (UniqueName: \"kubernetes.io/projected/7a901c1f-0f50-4d5e-94e8-03326a1c650f-kube-api-access-rt658\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049286 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049329 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049356 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-system-cni-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049383 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-default-certificate\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049409 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-etc-kubernetes\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049436 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-tuned\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049456 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-daemon-config\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049461 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598sq\" (UniqueName: \"kubernetes.io/projected/8f6d982a-e0d9-42ee-b342-290a02120b36-kube-api-access-598sq\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:41.049466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049484 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-kubelet\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049516 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-kubernetes\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049520 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-tuned\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049544 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049607 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bd5\" (UniqueName: \"kubernetes.io/projected/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-kube-api-access-f6bd5\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049633 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-device-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049659 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-default-certificate\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049675 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049699 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cnibin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049714 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-systemd\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049734 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b220d197-999f-4b20-af10-f9ffb9420614-crio-socket\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049764 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049786 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72732eb-0380-4f65-ab26-a788f64ad3fd-tmp-dir\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049821 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-hostroot\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049847 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-run\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049877 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d72732eb-0380-4f65-ab26-a788f64ad3fd-tmp-dir\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049906 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-agent-certs\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049926 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovn-node-metrics-cert\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049956 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-multus\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.049994 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-kubelet-config\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050070 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050091 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-konnectivity-ca\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050107 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-sys-fs\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050117 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff05d7d5-9766-49fb-86de-0c02df3dc0af-ovn-node-metrics-cert\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050123 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050173 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j892r\" (UniqueName: \"kubernetes.io/projected/8b24f7e8-b574-4609-96e5-9908302ac283-kube-api-access-j892r\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050194 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-bin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050219 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9662ce2-6f58-4385-8f20-f9ed018fea61-serving-cert\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050256 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cea9e9fb-3362-48cf-b2ec-3db89f981464-serving-cert\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050294 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-agent-certs\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050293 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-ovn\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050327 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e01f94e8-cf07-45d8-b023-0f87e7d41cdf-konnectivity-ca\") pod \"konnectivity-agent-wvndh\" (UID: \"e01f94e8-cf07-45d8-b023-0f87e7d41cdf\") " pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050354 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050383 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqng2\" (UniqueName: \"kubernetes.io/projected/3be78730-9048-43e9-9d31-e3819027d095-kube-api-access-kqng2\") pod \"kas-connection-checker-685895c7c9-5qrs6\" (UID: \"3be78730-9048-43e9-9d31-e3819027d095\") " pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050413 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9662ce2-6f58-4385-8f20-f9ed018fea61-serving-cert\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050413 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cea9e9fb-3362-48cf-b2ec-3db89f981464-serving-cert\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050419 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-tmp\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050458 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zldzc\" (UniqueName: \"kubernetes.io/projected/85d4d845-68cc-44e8-9503-5969ed5fcc1b-kube-api-access-zldzc\") pod \"kas-connection-checker-685895c7c9-g4sxx\" (UID: \"85d4d845-68cc-44e8-9503-5969ed5fcc1b\") " pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050467 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f440e8a-b448-4d18-9306-c5627dd6f5ad-tmp\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050478 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-dbus\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050502 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050528 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-conf-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050556 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vfx8\" (UniqueName: \"kubernetes.io/projected/a9662ce2-6f58-4385-8f20-f9ed018fea61-kube-api-access-8vfx8\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050580 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a515047-c16a-4352-acaf-16b5bd0dea55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050597 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-slash\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050613 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050638 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zj9w\" (UniqueName: \"kubernetes.io/projected/f237b168-098b-4df7-a56e-f245691a8086-kube-api-access-8zj9w\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050671 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cni-binary-copy\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050692 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgh4\" (UniqueName: \"kubernetes.io/projected/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04-kube-api-access-vsgh4\") pod \"network-check-source-6f78d59f99-9bwcj\" (UID: \"4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050710 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050724 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a515047-c16a-4352-acaf-16b5bd0dea55-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050797 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050824 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d72732eb-0380-4f65-ab26-a788f64ad3fd-hosts-file\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:41.050836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050852 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-stats-auth\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050870 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbwmh\" (UniqueName: \"kubernetes.io/projected/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-kube-api-access-gbwmh\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050886 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-node-log\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050898 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050903 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050904 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cni-binary-copy\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050947 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-conf\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.050979 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qd88\" (UniqueName: \"kubernetes.io/projected/8a515047-c16a-4352-acaf-16b5bd0dea55-kube-api-access-4qd88\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051000 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfn8\" (UniqueName: \"kubernetes.io/projected/cea9e9fb-3362-48cf-b2ec-3db89f981464-kube-api-access-xmfn8\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051061 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051165 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051201 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b220d197-999f-4b20-af10-f9ffb9420614-data-volume\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051206 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-stats-auth\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051267 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b220d197-999f-4b20-af10-f9ffb9420614-data-volume\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051271 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cnibin\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.052647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.051326 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a31617-3eb9-499e-a192-f4d24a34e835-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.053691 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.053673 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vxv\" (UniqueName: \"kubernetes.io/projected/ff05d7d5-9766-49fb-86de-0c02df3dc0af-kube-api-access-49vxv\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.053824 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.053801 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8zr\" (UniqueName: \"kubernetes.io/projected/f1f800e1-24b3-4323-97f3-2009efb63d1b-kube-api-access-bw8zr\") pod \"volume-data-source-validator-58c9f5f46d-xs4tv\" (UID: \"f1f800e1-24b3-4323-97f3-2009efb63d1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xs4tv" Nov 01 15:47:41.053987 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.053971 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4mn\" (UniqueName: \"kubernetes.io/projected/0e085d33-5390-4d2e-a616-61160ee87e46-kube-api-access-pw4mn\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.054502 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.054486 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rj4g\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.056335 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.056319 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vwg\" (UniqueName: \"kubernetes.io/projected/09cf87aa-e08e-4890-acf5-c8da31cd08da-kube-api-access-t4vwg\") pod \"network-check-target-bdfqx\" (UID: \"09cf87aa-e08e-4890-acf5-c8da31cd08da\") " pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:41.058072 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058048 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4tt\" (UniqueName: \"kubernetes.io/projected/1a7cd5bd-620b-4d98-b64a-4582788161a4-kube-api-access-rj4tt\") pod \"kas-connection-checker-685895c7c9-6v9fs\" (UID: \"1a7cd5bd-620b-4d98-b64a-4582788161a4\") " pod="kube-system/kas-connection-checker-685895c7c9-6v9fs" Nov 01 15:47:41.058161 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058089 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.058161 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058150 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhmx\" (UniqueName: \"kubernetes.io/projected/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-kube-api-access-czhmx\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:41.058256 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058174 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mk5k\" (UniqueName: \"kubernetes.io/projected/d72732eb-0380-4f65-ab26-a788f64ad3fd-kube-api-access-7mk5k\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:41.058305 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058274 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.058566 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058544 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k9p\" (UniqueName: \"kubernetes.io/projected/b220d197-999f-4b20-af10-f9ffb9420614-kube-api-access-c9k9p\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.058674 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058553 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-598sq\" (UniqueName: \"kubernetes.io/projected/8f6d982a-e0d9-42ee-b342-290a02120b36-kube-api-access-598sq\") pod \"ingress-canary-cf2mk\" (UID: \"8f6d982a-e0d9-42ee-b342-290a02120b36\") " pod="openshift-ingress-canary/ingress-canary-cf2mk" Nov 01 15:47:41.058841 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058819 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j892r\" (UniqueName: \"kubernetes.io/projected/8b24f7e8-b574-4609-96e5-9908302ac283-kube-api-access-j892r\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:41.058914 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058819 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt658\" (UniqueName: \"kubernetes.io/projected/7a901c1f-0f50-4d5e-94e8-03326a1c650f-kube-api-access-rt658\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.058914 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058829 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzh9b\" (UniqueName: \"kubernetes.io/projected/5671cc22-cecc-4d28-9054-9eee4e8dae58-kube-api-access-pzh9b\") pod \"migrator-66df5d496b-mz2hx\" (UID: \"5671cc22-cecc-4d28-9054-9eee4e8dae58\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mz2hx" Nov 01 15:47:41.059026 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.058965 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m465\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.059258 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.059239 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bd5\" (UniqueName: \"kubernetes.io/projected/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-kube-api-access-f6bd5\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.059504 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.059453 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qd88\" (UniqueName: \"kubernetes.io/projected/8a515047-c16a-4352-acaf-16b5bd0dea55-kube-api-access-4qd88\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-qvm99\" (UID: \"8a515047-c16a-4352-acaf-16b5bd0dea55\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" Nov 01 15:47:41.059601 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.059591 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcswr\" (UniqueName: \"kubernetes.io/projected/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-kube-api-access-gcswr\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.059729 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.059710 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm7b\" (UniqueName: \"kubernetes.io/projected/23a31617-3eb9-499e-a192-f4d24a34e835-kube-api-access-gvm7b\") pod \"insights-operator-7c4986d449-fz7xr\" (UID: \"23a31617-3eb9-499e-a192-f4d24a34e835\") " pod="openshift-insights/insights-operator-7c4986d449-fz7xr" Nov 01 15:47:41.059761 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.059710 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48zp\" (UniqueName: \"kubernetes.io/projected/57440ce3-722e-4835-927a-e8f2fac8b3c7-kube-api-access-p48zp\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:41.059761 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.059741 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsgh4\" (UniqueName: \"kubernetes.io/projected/4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04-kube-api-access-vsgh4\") pod \"network-check-source-6f78d59f99-9bwcj\" (UID: \"4adcbe8f-dcd5-47d5-bd8a-68f0206a7d04\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9bwcj" Nov 01 15:47:41.060125 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.060103 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zj9w\" (UniqueName: \"kubernetes.io/projected/f237b168-098b-4df7-a56e-f245691a8086-kube-api-access-8zj9w\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:41.060219 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.060162 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqng2\" (UniqueName: \"kubernetes.io/projected/3be78730-9048-43e9-9d31-e3819027d095-kube-api-access-kqng2\") pod \"kas-connection-checker-685895c7c9-5qrs6\" (UID: \"3be78730-9048-43e9-9d31-e3819027d095\") " pod="kube-system/kas-connection-checker-685895c7c9-5qrs6" Nov 01 15:47:41.060477 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.060461 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zldzc\" (UniqueName: \"kubernetes.io/projected/85d4d845-68cc-44e8-9503-5969ed5fcc1b-kube-api-access-zldzc\") pod \"kas-connection-checker-685895c7c9-g4sxx\" (UID: \"85d4d845-68cc-44e8-9503-5969ed5fcc1b\") " pod="kube-system/kas-connection-checker-685895c7c9-g4sxx" Nov 01 15:47:41.060555 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.060539 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfn8\" (UniqueName: \"kubernetes.io/projected/cea9e9fb-3362-48cf-b2ec-3db89f981464-kube-api-access-xmfn8\") pod \"console-operator-7cd56b8dbc-5qhqp\" (UID: \"cea9e9fb-3362-48cf-b2ec-3db89f981464\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:41.064827 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.061333 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227hr\" (UniqueName: \"kubernetes.io/projected/2f440e8a-b448-4d18-9306-c5627dd6f5ad-kube-api-access-227hr\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.064827 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.061875 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbwmh\" (UniqueName: \"kubernetes.io/projected/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-kube-api-access-gbwmh\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.064827 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.061906 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vgf\" (UniqueName: \"kubernetes.io/projected/7039a946-2ec8-414d-9093-7f355172fba4-kube-api-access-x9vgf\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.064827 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.063534 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vfx8\" (UniqueName: \"kubernetes.io/projected/a9662ce2-6f58-4385-8f20-f9ed018fea61-kube-api-access-8vfx8\") pod \"service-ca-operator-544d749c8c-wwww7\" (UID: \"a9662ce2-6f58-4385-8f20-f9ed018fea61\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" Nov 01 15:47:41.151694 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151666 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.151694 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151698 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-host\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.151891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151715 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-host-slash\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:41.151891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151730 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-systemd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.151891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151792 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-host\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.151891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151808 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff-host-slash\") pod \"iptables-alerter-9x9k2\" (UID: \"db4e2703-cec6-43fd-a0c0-b6ce1c4e3bff\") " pod="openshift-network-operator/iptables-alerter-9x9k2" Nov 01 15:47:41.151891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151839 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-systemd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.151891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151860 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-etc-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.151891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151879 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151895 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151910 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-lib-modules\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151926 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-k8s-cni-cncf-io\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151935 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-etc-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151942 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-socket-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151986 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-sys\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.151995 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-k8s-cni-cncf-io\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152016 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152063 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-systemd-units\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152069 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-sys\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152080 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-lib-modules\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152087 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.652064116 +0000 UTC m=+1.796518759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152099 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-systemd-units\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152100 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152125 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152128 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-socket-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152131 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-netd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152144 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152159 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-netd\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152163 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-system-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152187 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.652170686 +0000 UTC m=+1.796625326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152217 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-system-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.152222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152225 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-var-lib-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152249 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-multus-certs\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152264 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-var-lib-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152274 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152291 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-multus-certs\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152300 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152322 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-system-cni-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152335 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152348 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-etc-kubernetes\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152361 4911 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152371 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-system-cni-dir\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152371 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-kubelet\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152402 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-kubernetes\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152406 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-etc-kubernetes\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152418 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.652403916 +0000 UTC m=+1.796858555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152434 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-kubelet\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152452 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-kubernetes\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152458 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152491 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-device-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152480 4911 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152514 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152560 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-device-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152575 4911 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152598 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cnibin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152626 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-systemd\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152641 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls podName:57440ce3-722e-4835-927a-e8f2fac8b3c7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.652625887 +0000 UTC m=+1.797080520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-59xc9" (UID: "57440ce3-722e-4835-927a-e8f2fac8b3c7") : secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152668 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b220d197-999f-4b20-af10-f9ffb9420614-crio-socket\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152672 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-systemd\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152681 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-cnibin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152697 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152727 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b220d197-999f-4b20-af10-f9ffb9420614-crio-socket\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152729 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-hostroot\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152763 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-run\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152777 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152785 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-hostroot\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152790 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152821 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-multus\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152832 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-run\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152786 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-multus\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.152866 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.652851675 +0000 UTC m=+1.797306309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152893 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-kubelet-config\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152920 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-sys-fs\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152943 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152968 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-bin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152970 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-kubelet-config\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152997 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b220d197-999f-4b20-af10-f9ffb9420614-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.152993 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-ovn\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153057 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-ovn\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153061 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-cni-bin\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153069 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-dbus\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153108 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.153118 4911 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153069 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-sys-fs\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153131 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-conf-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.153157 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.653144073 +0000 UTC m=+1.797598709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153168 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153174 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-conf-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153175 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-slash\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153119 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337a97-e8fa-4061-ba5f-fea449c7d390-dbus\") pod \"global-pull-secret-syncer-xvq6f\" (UID: \"e9337a97-e8fa-4061-ba5f-fea449c7d390\") " pod="kube-system/global-pull-secret-syncer-xvq6f" Nov 01 15:47:41.153176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153207 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d72732eb-0380-4f65-ab26-a788f64ad3fd-hosts-file\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153246 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-node-log\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153270 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153285 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d72732eb-0380-4f65-ab26-a788f64ad3fd-hosts-file\") pod \"node-resolver-wx7cw\" (UID: \"d72732eb-0380-4f65-ab26-a788f64ad3fd\") " pod="openshift-dns/node-resolver-wx7cw" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153293 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-conf\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153204 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-slash\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153305 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-node-log\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153322 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153349 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cnibin\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153372 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f237b168-098b-4df7-a56e-f245691a8086-host\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153412 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153419 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153430 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysctl-conf\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153420 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a3d24bb-ebe0-4623-88d7-77cd90a8b142-cnibin\") pod \"multus-additional-cni-plugins-m8qrf\" (UID: \"1a3d24bb-ebe0-4623-88d7-77cd90a8b142\") " pod="openshift-multus/multus-additional-cni-plugins-m8qrf" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.153446 4911 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153448 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f237b168-098b-4df7-a56e-f245691a8086-host\") pod \"node-ca-czbft\" (UID: \"f237b168-098b-4df7-a56e-f245691a8086\") " pod="openshift-image-registry/node-ca-czbft" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.153478 4911 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153485 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-netns\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153448 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-run-netns\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.153502 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert podName:19f38b5c-f14d-48fd-b0cf-a2581c62384a nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.65348921 +0000 UTC m=+1.797943848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j46fh" (UID: "19f38b5c-f14d-48fd-b0cf-a2581c62384a") : secret "networking-console-plugin-cert" not found Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.153528 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:41.653521835 +0000 UTC m=+1.797976454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153540 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153557 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153574 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-socket-dir-parent\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153574 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153588 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-kubelet\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153605 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-bin\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153619 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153625 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153626 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-socket-dir-parent\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153659 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-var-lib-kubelet\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153670 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-host-cni-bin\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153663 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysconfig\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153685 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-sysconfig\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153708 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-modprobe-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153733 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-var-lib-kubelet\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153757 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-registration-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153783 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153807 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-var-lib-kubelet\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153808 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153826 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7039a946-2ec8-414d-9093-7f355172fba4-registration-dir\") pod \"aws-ebs-csi-driver-node-jp69b\" (UID: \"7039a946-2ec8-414d-9093-7f355172fba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp69b" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153837 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-run-openvswitch\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153845 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-log-socket\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153847 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f440e8a-b448-4d18-9306-c5627dd6f5ad-etc-modprobe-d\") pod \"tuned-v77kh\" (UID: \"2f440e8a-b448-4d18-9306-c5627dd6f5ad\") " pod="openshift-cluster-node-tuning-operator/tuned-v77kh" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153870 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-netns\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153891 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-multus-cni-dir\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153900 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff05d7d5-9766-49fb-86de-0c02df3dc0af-log-socket\") pod \"ovnkube-node-hvbbz\" (UID: \"ff05d7d5-9766-49fb-86de-0c02df3dc0af\") " pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.153945 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a901c1f-0f50-4d5e-94e8-03326a1c650f-host-run-netns\") pod \"multus-zn8p6\" (UID: \"7a901c1f-0f50-4d5e-94e8-03326a1c650f\") " pod="openshift-multus/multus-zn8p6" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.155060 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f70b67b3-d8eb-4065-8650-b26de9bc9f7c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5rr7\" (UID: \"f70b67b3-d8eb-4065-8650-b26de9bc9f7c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.156355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.155614 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b24f7e8-b574-4609-96e5-9908302ac283-metrics-certs\") pod \"network-metrics-daemon-l6qq5\" (UID: \"8b24f7e8-b574-4609-96e5-9908302ac283\") " pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:41.235468 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.235444 4911 scope.go:122] "RemoveContainer" containerID="f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66" Nov 01 15:47:41.236472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.236447 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" Nov 01 15:47:41.241844 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.241756 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6qq5" Nov 01 15:47:41.546023 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.545997 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7"] Nov 01 15:47:41.595271 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.595246 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6qq5"] Nov 01 15:47:41.598115 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:41.598087 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b24f7e8_b574_4609_96e5_9908302ac283.slice/crio-d33c09ecc2793f54ea60d43466bbe942cfd0b6f9fe85df5dc0927e04f48b4b2f WatchSource:0}: Error finding container d33c09ecc2793f54ea60d43466bbe942cfd0b6f9fe85df5dc0927e04f48b4b2f: Status 404 returned error can't find the container with id d33c09ecc2793f54ea60d43466bbe942cfd0b6f9fe85df5dc0927e04f48b4b2f Nov 01 15:47:41.600225 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.600210 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:47:41.658839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.658809 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:41.658976 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.658849 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:41.658976 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.658876 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:41.659124 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.658982 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:41.659124 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659000 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:41.659124 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659057 4911 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:41.659124 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.658982 4911 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:41.659124 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.659084 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:41.659124 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659106 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.659085875 +0000 UTC m=+2.803540510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:41.659124 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659129 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls podName:57440ce3-722e-4835-927a-e8f2fac8b3c7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.659119736 +0000 UTC m=+2.803574360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-59xc9" (UID: "57440ce3-722e-4835-927a-e8f2fac8b3c7") : secret "cluster-monitoring-operator-tls" not found Nov 01 15:47:41.659352 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659140 4911 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:41.659352 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659146 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.659138169 +0000 UTC m=+2.803592788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:41.659352 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659178 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.659166744 +0000 UTC m=+2.803621398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:41.659352 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.659223 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:41.659352 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.659255 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.659352 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.659308 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:41.659352 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.659326 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:41.659637 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659414 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.659405908 +0000 UTC m=+2.803860527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:41.659637 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659411 4911 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:41.659637 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659444 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:41.659637 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659459 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:41.659637 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659460 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.65944705 +0000 UTC m=+2.803901683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:41.659637 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:41.659522 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.659507641 +0000 UTC m=+2.803962268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:41.661650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.661634 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19f38b5c-f14d-48fd-b0cf-a2581c62384a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j46fh\" (UID: \"19f38b5c-f14d-48fd-b0cf-a2581c62384a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:41.838630 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:41.838558 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" Nov 01 15:47:42.015656 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.015628 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/2.log" Nov 01 15:47:42.016053 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.016015 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/1.log" Nov 01 15:47:42.016097 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.016064 4911 generic.go:356] "Generic (PLEG): container finished" podID="cea9e9fb-3362-48cf-b2ec-3db89f981464" containerID="4d6f9d7d724aefe91ca790228ceed6509ff568e257552ec1560a8390697748a9" exitCode=255 Nov 01 15:47:42.016158 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.016140 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" event={"ID":"cea9e9fb-3362-48cf-b2ec-3db89f981464","Type":"ContainerDied","Data":"4d6f9d7d724aefe91ca790228ceed6509ff568e257552ec1560a8390697748a9"} Nov 01 15:47:42.016208 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.016198 4911 scope.go:122] "RemoveContainer" containerID="f06b91c8f7d3791e548673cc9da25e62510a3ac0483fc1036ee52e99e6bb5b66" Nov 01 15:47:42.016554 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.016535 4911 scope.go:122] "RemoveContainer" containerID="4d6f9d7d724aefe91ca790228ceed6509ff568e257552ec1560a8390697748a9" Nov 01 15:47:42.016787 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.016765 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:42.017443 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.017422 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" event={"ID":"f70b67b3-d8eb-4065-8650-b26de9bc9f7c","Type":"ContainerStarted","Data":"2bd7213271839a8119c8f46d7bdd96947da70ab1eb65704dd2f403e1349bde4c"} Nov 01 15:47:42.018595 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.018566 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6qq5" event={"ID":"8b24f7e8-b574-4609-96e5-9908302ac283","Type":"ContainerStarted","Data":"d33c09ecc2793f54ea60d43466bbe942cfd0b6f9fe85df5dc0927e04f48b4b2f"} Nov 01 15:47:42.125729 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.125665 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j46fh"] Nov 01 15:47:42.128703 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:42.128674 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f38b5c_f14d_48fd_b0cf_a2581c62384a.slice/crio-4a990ce4c832606059ee946ee9f9642063ed03bd5648e43b127cdc8d6c09f0d8 WatchSource:0}: Error finding container 4a990ce4c832606059ee946ee9f9642063ed03bd5648e43b127cdc8d6c09f0d8: Status 404 returned error can't find the container with id 4a990ce4c832606059ee946ee9f9642063ed03bd5648e43b127cdc8d6c09f0d8 Nov 01 15:47:42.664841 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.664802 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:47:42.670215 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.670183 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:42.670381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.670234 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:42.670381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.670273 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:42.670381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.670290 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:42.670381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.670328 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:42.670381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.670363 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:42.670632 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.670384 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:42.670632 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670475 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:42.670632 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670485 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:42.670632 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670531 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.670514822 +0000 UTC m=+4.814969458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:42.670632 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670566 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:42.670632 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670581 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:42.670632 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670617 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.670605875 +0000 UTC m=+4.815060512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:42.670943 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670784 4911 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:42.670943 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670826 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.670813 +0000 UTC m=+4.815267633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:42.670943 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670847 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.670838508 +0000 UTC m=+4.815293130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:42.670943 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670883 4911 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:42.670943 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670895 4911 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:42.670943 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670916 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.670906125 +0000 UTC m=+4.815360747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:42.670943 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:42.670931 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.670923261 +0000 UTC m=+4.815377879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:42.673754 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.673728 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57440ce3-722e-4835-927a-e8f2fac8b3c7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-59xc9\" (UID: \"57440ce3-722e-4835-927a-e8f2fac8b3c7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:42.734819 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:42.734780 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" Nov 01 15:47:43.020346 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:43.020317 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9"] Nov 01 15:47:43.024967 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:43.024947 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/2.log" Nov 01 15:47:43.026332 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:43.026303 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" event={"ID":"19f38b5c-f14d-48fd-b0cf-a2581c62384a","Type":"ContainerStarted","Data":"4a990ce4c832606059ee946ee9f9642063ed03bd5648e43b127cdc8d6c09f0d8"} Nov 01 15:47:43.159408 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:43.159276 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57440ce3_722e_4835_927a_e8f2fac8b3c7.slice/crio-5310c1b21622d7872a5ef0dbe1fbfbe2cce33cf2b93bfb791cd3d010ecda7e84 WatchSource:0}: Error finding container 5310c1b21622d7872a5ef0dbe1fbfbe2cce33cf2b93bfb791cd3d010ecda7e84: Status 404 returned error can't find the container with id 5310c1b21622d7872a5ef0dbe1fbfbe2cce33cf2b93bfb791cd3d010ecda7e84 Nov 01 15:47:44.032877 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.032833 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6qq5" event={"ID":"8b24f7e8-b574-4609-96e5-9908302ac283","Type":"ContainerStarted","Data":"bdd22322157550adf5fe25ab19c7fd42309559e259d6085b034fb4e6b7aaf88b"} Nov 01 15:47:44.033366 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.032900 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6qq5" event={"ID":"8b24f7e8-b574-4609-96e5-9908302ac283","Type":"ContainerStarted","Data":"5181414065d1e1a70189045c9f78b9bf16e304d71b2483de5a17fd60a6124393"} Nov 01 15:47:44.034669 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.034644 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" event={"ID":"57440ce3-722e-4835-927a-e8f2fac8b3c7","Type":"ContainerStarted","Data":"5310c1b21622d7872a5ef0dbe1fbfbe2cce33cf2b93bfb791cd3d010ecda7e84"} Nov 01 15:47:44.689213 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.689115 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:44.689213 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.689202 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.689236 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689301 4911 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689354 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689367 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:48.689353709 +0000 UTC m=+8.833808332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : configmap references non-existent config key: service-ca.crt Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689371 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84cf9dbdbb-c9jvz: secret "image-registry-tls" not found Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689389 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs podName:e3d66b38-3dfa-46eb-a2c7-d8a5de28be89 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:48.689378301 +0000 UTC m=+8.833832919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs") pod "router-default-69f6694997-vvcm2" (UID: "e3d66b38-3dfa-46eb-a2c7-d8a5de28be89") : secret "router-metrics-certs-default" not found Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689419 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls podName:3eab1985-6883-4db6-ac14-0d3ae3159c20 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:48.689400125 +0000 UTC m=+8.833854746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls") pod "image-registry-84cf9dbdbb-c9jvz" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20") : secret "image-registry-tls" not found Nov 01 15:47:44.689440 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.689439 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.689485 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:44.689519 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689582 4911 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689626 4911 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689633 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:48.689618717 +0000 UTC m=+8.834073351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689640 4911 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689658 4911 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fb79fc6b7-gq4fr: secret "image-registry-tls" not found Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689659 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls podName:0e085d33-5390-4d2e-a616-61160ee87e46 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:48.689650846 +0000 UTC m=+8.834105465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls") pod "dns-default-jx495" (UID: "0e085d33-5390-4d2e-a616-61160ee87e46") : secret "dns-default-metrics-tls" not found Nov 01 15:47:44.689806 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:44.689716 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls podName:5432881f-c368-4d90-b8d2-4958ba502338 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:48.689701891 +0000 UTC m=+8.834156513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls") pod "image-registry-6fb79fc6b7-gq4fr" (UID: "5432881f-c368-4d90-b8d2-4958ba502338") : secret "image-registry-tls" not found Nov 01 15:47:45.040982 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.040944 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" event={"ID":"f70b67b3-d8eb-4065-8650-b26de9bc9f7c","Type":"ContainerStarted","Data":"5e752983935d1e4db42fd104169bcae95cd2dcc8c1a17c4ef28c056c6aaa9645"} Nov 01 15:47:45.041531 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.041066 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5rr7" event={"ID":"f70b67b3-d8eb-4065-8650-b26de9bc9f7c","Type":"ContainerStarted","Data":"62da74a0b665bdaa74760fd11eb91e9515c8ae86a5dfa4dbf1849ad824842ad5"} Nov 01 15:47:45.042934 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.042908 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j46fh" event={"ID":"19f38b5c-f14d-48fd-b0cf-a2581c62384a","Type":"ContainerStarted","Data":"33a050bd0500b2e3d4a38ca3d451493eb8539c62ec0d0c8ca95b5fa6a99b5ac0"} Nov 01 15:47:45.044732 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.044703 4911 generic.go:356] "Generic (PLEG): container finished" podID="8a515047-c16a-4352-acaf-16b5bd0dea55" containerID="86c05e813218aeb00b8629ba3257f94b0de50f1181136f6a6d6d2161f00057b5" exitCode=0 Nov 01 15:47:45.044856 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.044781 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" event={"ID":"8a515047-c16a-4352-acaf-16b5bd0dea55","Type":"ContainerDied","Data":"86c05e813218aeb00b8629ba3257f94b0de50f1181136f6a6d6d2161f00057b5"} Nov 01 15:47:45.045228 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.045208 4911 scope.go:122] "RemoveContainer" containerID="86c05e813218aeb00b8629ba3257f94b0de50f1181136f6a6d6d2161f00057b5" Nov 01 15:47:45.420576 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.420551 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wx7cw_d72732eb-0380-4f65-ab26-a788f64ad3fd/dns-node-resolver/0.log" Nov 01 15:47:45.556013 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.555971 4911 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:45.556467 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:45.556450 4911 scope.go:122] "RemoveContainer" containerID="4d6f9d7d724aefe91ca790228ceed6509ff568e257552ec1560a8390697748a9" Nov 01 15:47:45.556697 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:45.556673 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:46.049526 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:46.049493 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-59xc9" event={"ID":"57440ce3-722e-4835-927a-e8f2fac8b3c7","Type":"ContainerStarted","Data":"33592c56b954e006910099b7440c2401190114b1338f4b1833e74116fa94a4ac"} Nov 01 15:47:46.051366 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:46.051339 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-qvm99" event={"ID":"8a515047-c16a-4352-acaf-16b5bd0dea55","Type":"ContainerStarted","Data":"3318a5e57c9230294f691c0409a43f86acef734de84accb9605e69f222f17696"} Nov 01 15:47:46.221280 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:46.221253 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-czbft_f237b168-098b-4df7-a56e-f245691a8086/node-ca/0.log" Nov 01 15:47:47.021420 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:47.021393 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cf2mk_8f6d982a-e0d9-42ee-b342-290a02120b36/serve-healthcheck-canary/0.log" Nov 01 15:47:47.421079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:47.420997 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-mz2hx_5671cc22-cecc-4d28-9054-9eee4e8dae58/migrator/0.log" Nov 01 15:47:47.512485 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:47.512461 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:47.620970 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:47.620940 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-qvm99_8a515047-c16a-4352-acaf-16b5bd0dea55/kube-storage-version-migrator-operator/1.log" Nov 01 15:47:48.507176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.507148 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:47:48.507627 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.507365 4911 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:48.628616 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.628560 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:48.629097 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.629077 4911 scope.go:122] "RemoveContainer" containerID="4d6f9d7d724aefe91ca790228ceed6509ff568e257552ec1560a8390697748a9" Nov 01 15:47:48.629266 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:48.629248 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:48.729712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.729676 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:48.729894 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.729763 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:48.729894 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.729843 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:48.729894 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.729867 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:48.730081 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.729928 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:48.730081 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.729999 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:48.730081 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:48.730004 4911 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:48.730239 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:48.730139 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls podName:b220d197-999f-4b20-af10-f9ffb9420614 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:56.730118961 +0000 UTC m=+16.874573585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x8kz" (UID: "b220d197-999f-4b20-af10-f9ffb9420614") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:48.730858 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.730832 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-service-ca-bundle\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:48.732450 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.732424 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d66b38-3dfa-46eb-a2c7-d8a5de28be89-metrics-certs\") pod \"router-default-69f6694997-vvcm2\" (UID: \"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89\") " pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:48.732571 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.732471 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"image-registry-6fb79fc6b7-gq4fr\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:48.732700 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.732679 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e085d33-5390-4d2e-a616-61160ee87e46-metrics-tls\") pod \"dns-default-jx495\" (UID: \"0e085d33-5390-4d2e-a616-61160ee87e46\") " pod="openshift-dns/dns-default-jx495" Nov 01 15:47:48.732757 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.732704 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"image-registry-84cf9dbdbb-c9jvz\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:48.734621 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.734597 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:48.737105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.737086 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:48.750539 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.750512 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jx495" Nov 01 15:47:48.750874 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:48.750828 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:49.024882 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.024807 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69f6694997-vvcm2"] Nov 01 15:47:49.027926 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.027885 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fb79fc6b7-gq4fr"] Nov 01 15:47:49.028229 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:49.028204 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d66b38_3dfa_46eb_a2c7_d8a5de28be89.slice/crio-fa485295a34869a24b54822e207ba0355f632dde9760b06eb1b695182694bf4b WatchSource:0}: Error finding container fa485295a34869a24b54822e207ba0355f632dde9760b06eb1b695182694bf4b: Status 404 returned error can't find the container with id fa485295a34869a24b54822e207ba0355f632dde9760b06eb1b695182694bf4b Nov 01 15:47:49.030808 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:49.030777 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5432881f_c368_4d90_b8d2_4958ba502338.slice/crio-caba6031fd1615757d387354db452f41195defb5006682def16acfd3b230c0c7 WatchSource:0}: Error finding container caba6031fd1615757d387354db452f41195defb5006682def16acfd3b230c0c7: Status 404 returned error can't find the container with id caba6031fd1615757d387354db452f41195defb5006682def16acfd3b230c0c7 Nov 01 15:47:49.049149 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.049122 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jx495"] Nov 01 15:47:49.051283 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.051261 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84cf9dbdbb-c9jvz"] Nov 01 15:47:49.054201 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:49.054173 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e085d33_5390_4d2e_a616_61160ee87e46.slice/crio-8531fde3a380d3404acea66c36e8f90edeee37ac6cb95f1fdcbda882179cb889 WatchSource:0}: Error finding container 8531fde3a380d3404acea66c36e8f90edeee37ac6cb95f1fdcbda882179cb889: Status 404 returned error can't find the container with id 8531fde3a380d3404acea66c36e8f90edeee37ac6cb95f1fdcbda882179cb889 Nov 01 15:47:49.054659 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:49.054633 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eab1985_6883_4db6_ac14_0d3ae3159c20.slice/crio-c19b28c4d6d545663a035c94f2e197faa10ad76647572fa396dc49fcec441112 WatchSource:0}: Error finding container c19b28c4d6d545663a035c94f2e197faa10ad76647572fa396dc49fcec441112: Status 404 returned error can't find the container with id c19b28c4d6d545663a035c94f2e197faa10ad76647572fa396dc49fcec441112 Nov 01 15:47:49.074069 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.074018 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jx495" event={"ID":"0e085d33-5390-4d2e-a616-61160ee87e46","Type":"ContainerStarted","Data":"8531fde3a380d3404acea66c36e8f90edeee37ac6cb95f1fdcbda882179cb889"} Nov 01 15:47:49.075643 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.075614 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69f6694997-vvcm2" event={"ID":"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89","Type":"ContainerStarted","Data":"fa485295a34869a24b54822e207ba0355f632dde9760b06eb1b695182694bf4b"} Nov 01 15:47:49.077224 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.077194 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" event={"ID":"3eab1985-6883-4db6-ac14-0d3ae3159c20","Type":"ContainerStarted","Data":"c19b28c4d6d545663a035c94f2e197faa10ad76647572fa396dc49fcec441112"} Nov 01 15:47:49.078529 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:49.078501 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" event={"ID":"5432881f-c368-4d90-b8d2-4958ba502338","Type":"ContainerStarted","Data":"caba6031fd1615757d387354db452f41195defb5006682def16acfd3b230c0c7"} Nov 01 15:47:50.084369 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:50.084323 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69f6694997-vvcm2" event={"ID":"e3d66b38-3dfa-46eb-a2c7-d8a5de28be89","Type":"ContainerStarted","Data":"9c843eb9db8c0e75a8fdd390996670c7fa0e5566632acd46625645fc02a6321e"} Nov 01 15:47:50.086144 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:50.086108 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" event={"ID":"3eab1985-6883-4db6-ac14-0d3ae3159c20","Type":"ContainerStarted","Data":"cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de"} Nov 01 15:47:50.087836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:50.087814 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" event={"ID":"5432881f-c368-4d90-b8d2-4958ba502338","Type":"ContainerStarted","Data":"8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d"} Nov 01 15:47:50.735010 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:50.734972 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:50.737542 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:50.737518 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:50.773006 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:50.772960 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:50.774196 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:50.774176 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69f6694997-vvcm2" Nov 01 15:47:51.093610 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:51.093580 4911 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:51.093610 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:51.093580 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jx495" event={"ID":"0e085d33-5390-4d2e-a616-61160ee87e46","Type":"ContainerStarted","Data":"3489f53ece4751e74f3c17d423b1b8eda74f66099a0b28b0ab31402554c7af9e"} Nov 01 15:47:51.094086 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:51.093970 4911 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:51.094185 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:51.094163 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jx495" Nov 01 15:47:51.094229 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:51.094206 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jx495" event={"ID":"0e085d33-5390-4d2e-a616-61160ee87e46","Type":"ContainerStarted","Data":"19271f23c4bb5589f26d2b97fc12703775148fbdc372be55f311337aaa661169"} Nov 01 15:47:51.958715 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:51.958685 4911 scope.go:122] "RemoveContainer" containerID="4d6f9d7d724aefe91ca790228ceed6509ff568e257552ec1560a8390697748a9" Nov 01 15:47:52.098512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:52.098490 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 15:47:52.098867 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:52.098849 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/2.log" Nov 01 15:47:52.098917 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:52.098884 4911 generic.go:356] "Generic (PLEG): container finished" podID="cea9e9fb-3362-48cf-b2ec-3db89f981464" containerID="020720f03643b58b11ba9781f3b9a86204c5f2db10fe5fb52a96d23d3c1ac34e" exitCode=255 Nov 01 15:47:52.098996 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:52.098969 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" event={"ID":"cea9e9fb-3362-48cf-b2ec-3db89f981464","Type":"ContainerDied","Data":"020720f03643b58b11ba9781f3b9a86204c5f2db10fe5fb52a96d23d3c1ac34e"} Nov 01 15:47:52.099074 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:52.099051 4911 scope.go:122] "RemoveContainer" containerID="4d6f9d7d724aefe91ca790228ceed6509ff568e257552ec1560a8390697748a9" Nov 01 15:47:52.099553 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:52.099534 4911 scope.go:122] "RemoveContainer" containerID="020720f03643b58b11ba9781f3b9a86204c5f2db10fe5fb52a96d23d3c1ac34e" Nov 01 15:47:52.099765 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:52.099709 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:53.103830 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:53.103802 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 15:47:55.556492 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:55.556446 4911 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:55.556878 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:55.556823 4911 scope.go:122] "RemoveContainer" containerID="020720f03643b58b11ba9781f3b9a86204c5f2db10fe5fb52a96d23d3c1ac34e" Nov 01 15:47:55.557065 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:55.557022 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:47:56.682896 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:56.682861 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:47:56.774854 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:56.774805 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fb79fc6b7-gq4fr"] Nov 01 15:47:56.775384 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:56.775369 4911 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:56.799398 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:56.799359 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:56.801981 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:56.801950 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b220d197-999f-4b20-af10-f9ffb9420614-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x8kz\" (UID: \"b220d197-999f-4b20-af10-f9ffb9420614\") " pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:56.852721 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:56.852687 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6x8kz" Nov 01 15:47:57.127327 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:57.127296 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6x8kz"] Nov 01 15:47:57.130578 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:47:57.130547 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb220d197_999f_4b20_af10_f9ffb9420614.slice/crio-1201831d44d7b24c0573ac4c6da18e9b493b6ae9b3a63f96fd6243169230f8c0 WatchSource:0}: Error finding container 1201831d44d7b24c0573ac4c6da18e9b493b6ae9b3a63f96fd6243169230f8c0: Status 404 returned error can't find the container with id 1201831d44d7b24c0573ac4c6da18e9b493b6ae9b3a63f96fd6243169230f8c0 Nov 01 15:47:58.106175 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:58.106141 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:47:58.123099 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:58.123070 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x8kz" event={"ID":"b220d197-999f-4b20-af10-f9ffb9420614","Type":"ContainerStarted","Data":"1676f6ce37f91a384879e44d80f20c4690957c62df43b3543387d02f344dac58"} Nov 01 15:47:58.123249 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:58.123123 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x8kz" event={"ID":"b220d197-999f-4b20-af10-f9ffb9420614","Type":"ContainerStarted","Data":"480536134e68bacc7c2859255ecf01b8e264397f002537f064e9e44701f0af3f"} Nov 01 15:47:58.123249 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:58.123145 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x8kz" event={"ID":"b220d197-999f-4b20-af10-f9ffb9420614","Type":"ContainerStarted","Data":"1201831d44d7b24c0573ac4c6da18e9b493b6ae9b3a63f96fd6243169230f8c0"} Nov 01 15:47:58.629592 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:58.629562 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:47:58.629989 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:47:58.629973 4911 scope.go:122] "RemoveContainer" containerID="020720f03643b58b11ba9781f3b9a86204c5f2db10fe5fb52a96d23d3c1ac34e" Nov 01 15:47:58.630210 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:47:58.630177 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qhqp_openshift-console-operator(cea9e9fb-3362-48cf-b2ec-3db89f981464)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" podUID="cea9e9fb-3362-48cf-b2ec-3db89f981464" Nov 01 15:48:00.139960 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:00.139909 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x8kz" event={"ID":"b220d197-999f-4b20-af10-f9ffb9420614","Type":"ContainerStarted","Data":"da6407390c5d973060ff8cf8bf21171ef5e3995eafadb5f8b5c63985f608b0ee"} Nov 01 15:48:02.106872 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:02.106839 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jx495" Nov 01 15:48:03.071399 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:03.071370 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wvndh" Nov 01 15:48:04.086322 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.086290 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hvbbz" Nov 01 15:48:04.397204 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.397115 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wgx5w"] Nov 01 15:48:04.402296 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.402273 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.467723 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.467685 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z6wzh\"" Nov 01 15:48:04.467885 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.467747 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 15:48:04.467885 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.467802 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 15:48:04.501954 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.501932 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 15:48:04.501954 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.501932 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 15:48:04.561344 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561314 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncts\" (UniqueName: \"kubernetes.io/projected/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-kube-api-access-9ncts\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561511 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561360 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-textfile\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561511 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561380 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-root\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561511 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561478 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561511 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561499 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-wtmp\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561757 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561520 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-metrics-client-ca\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561757 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561536 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561757 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561560 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-sys\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.561757 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.561712 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-tls\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.662809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662719 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-textfile\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.662809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662756 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-root\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.662809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662785 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.662809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662805 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-wtmp\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662823 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-metrics-client-ca\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662839 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662861 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-root\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662872 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-sys\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662905 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-sys\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662939 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-tls\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662957 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-wtmp\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.662997 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncts\" (UniqueName: \"kubernetes.io/projected/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-kube-api-access-9ncts\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663181 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.663177 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-textfile\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663548 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.663524 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-metrics-client-ca\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.663587 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.663527 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.665265 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.665242 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-tls\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.665376 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.665300 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.670503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.670482 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncts\" (UniqueName: \"kubernetes.io/projected/ca68b7f9-2b1d-4b17-81e6-68c64ce8428d-kube-api-access-9ncts\") pod \"node-exporter-wgx5w\" (UID: \"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d\") " pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.718782 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:04.718749 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wgx5w" Nov 01 15:48:04.731415 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:48:04.731381 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca68b7f9_2b1d_4b17_81e6_68c64ce8428d.slice/crio-212d3e087bb2e1859f346a81a7a7881679d920a6ef703518a38dfb4ca2a9fec3 WatchSource:0}: Error finding container 212d3e087bb2e1859f346a81a7a7881679d920a6ef703518a38dfb4ca2a9fec3: Status 404 returned error can't find the container with id 212d3e087bb2e1859f346a81a7a7881679d920a6ef703518a38dfb4ca2a9fec3 Nov 01 15:48:05.164646 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:05.164609 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wgx5w" event={"ID":"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d","Type":"ContainerStarted","Data":"212d3e087bb2e1859f346a81a7a7881679d920a6ef703518a38dfb4ca2a9fec3"} Nov 01 15:48:06.170531 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:06.170448 4911 generic.go:356] "Generic (PLEG): container finished" podID="ca68b7f9-2b1d-4b17-81e6-68c64ce8428d" containerID="169748fd5819dfeb4dc7f30bd575fe11bdd8392017fcee53175f6d2dce35ac7d" exitCode=0 Nov 01 15:48:06.170880 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:06.170532 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wgx5w" event={"ID":"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d","Type":"ContainerDied","Data":"169748fd5819dfeb4dc7f30bd575fe11bdd8392017fcee53175f6d2dce35ac7d"} Nov 01 15:48:06.687974 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:06.687946 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:48:07.176468 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:07.176427 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wgx5w" event={"ID":"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d","Type":"ContainerStarted","Data":"d20d5829f00c233353d26c80b29043a8e23f2063061082cb79697df66ef04c41"} Nov 01 15:48:07.176868 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:07.176491 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wgx5w" event={"ID":"ca68b7f9-2b1d-4b17-81e6-68c64ce8428d","Type":"ContainerStarted","Data":"848c36fe6860cc80e2dc7fd78e7476fae87ed8794f583c1aef67b593dc8b46b9"} Nov 01 15:48:07.204136 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:07.203991 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wgx5w" podStartSLOduration=2.122172741 podStartE2EDuration="3.203417849s" totalImagesPullingTime="1.014896828s" totalInitContainerRuntime="66.34828ms" isStatefulPod=true podCreationTimestamp="2026-11-01 15:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:07.203337575 +0000 UTC m=+27.347792217" watchObservedRunningTime="2026-11-01 15:48:07.203417849 +0000 UTC m=+27.347872491" Nov 01 15:48:08.111513 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:08.111482 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:48:10.864456 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:48:10.864404 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:48:12.667747 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:12.667719 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bdfqx" Nov 01 15:48:17.958739 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:17.958701 4911 scope.go:122] "RemoveContainer" containerID="020720f03643b58b11ba9781f3b9a86204c5f2db10fe5fb52a96d23d3c1ac34e" Nov 01 15:48:18.222019 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:18.221941 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 15:48:18.222190 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:18.222027 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" event={"ID":"cea9e9fb-3362-48cf-b2ec-3db89f981464","Type":"ContainerStarted","Data":"a33748a72db006fffb7684a6245c709e00ed0eb4af6c2d8ffe96c744e910315c"} Nov 01 15:48:18.222342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:18.222323 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:48:18.400544 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:18.400515 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qhqp" Nov 01 15:48:18.924101 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:18.924064 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84cf9dbdbb-c9jvz"] Nov 01 15:48:21.800447 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:21.800361 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" podUID="5432881f-c368-4d90-b8d2-4958ba502338" containerName="registry" containerID="cri-o://8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d" gracePeriod=30 Nov 01 15:48:22.043622 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.043596 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:48:22.111851 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.111773 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.111851 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.111829 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.111851 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.111847 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.112164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.111871 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.112164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.111922 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.112164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.111970 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.112164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.112004 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.112164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.112059 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\") pod \"5432881f-c368-4d90-b8d2-4958ba502338\" (UID: \"5432881f-c368-4d90-b8d2-4958ba502338\") " Nov 01 15:48:22.112410 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.112280 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:22.112554 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.112507 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:22.114523 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.114491 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:22.114824 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.114803 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "kube-api-access-4m465". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:22.114824 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.114809 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:22.114964 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.114808 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:22.114964 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.114894 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:22.119233 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.119214 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted" pod "5432881f-c368-4d90-b8d2-4958ba502338" (UID: "5432881f-c368-4d90-b8d2-4958ba502338"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:48:22.213253 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213213 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m465\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-kube-api-access-4m465\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.213253 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213244 4911 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-trusted-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.213253 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213255 4911 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-registry-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.213481 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213265 4911 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5432881f-c368-4d90-b8d2-4958ba502338-registry-certificates\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.213481 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213274 4911 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-image-registry-private-configuration\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.213481 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213283 4911 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5432881f-c368-4d90-b8d2-4958ba502338-installation-pull-secrets\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.213481 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213295 4911 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5432881f-c368-4d90-b8d2-4958ba502338-bound-sa-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.213481 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.213304 4911 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5432881f-c368-4d90-b8d2-4958ba502338-ca-trust-extracted\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:22.237841 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.237805 4911 generic.go:356] "Generic (PLEG): container finished" podID="5432881f-c368-4d90-b8d2-4958ba502338" containerID="8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d" exitCode=0 Nov 01 15:48:22.237970 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.237854 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" Nov 01 15:48:22.237970 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.237888 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" event={"ID":"5432881f-c368-4d90-b8d2-4958ba502338","Type":"ContainerDied","Data":"8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d"} Nov 01 15:48:22.237970 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.237944 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fb79fc6b7-gq4fr" event={"ID":"5432881f-c368-4d90-b8d2-4958ba502338","Type":"ContainerDied","Data":"caba6031fd1615757d387354db452f41195defb5006682def16acfd3b230c0c7"} Nov 01 15:48:22.237970 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.237964 4911 scope.go:122] "RemoveContainer" containerID="8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d" Nov 01 15:48:22.246510 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.246492 4911 scope.go:122] "RemoveContainer" containerID="8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d" Nov 01 15:48:22.246794 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:48:22.246767 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d\": container with ID starting with 8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d not found: ID does not exist" containerID="8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d" Nov 01 15:48:22.246836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.246806 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d"} err="failed to get container status \"8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d\": rpc error: code = NotFound desc = could not find container \"8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d\": container with ID starting with 8e510dee5204e20ceb136d2ce5e43e4cd764d4634cb201bf6af8711b6442ea1d not found: ID does not exist" Nov 01 15:48:22.258873 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.258848 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fb79fc6b7-gq4fr"] Nov 01 15:48:22.262994 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:22.262974 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fb79fc6b7-gq4fr"] Nov 01 15:48:23.963472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:23.963436 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5432881f-c368-4d90-b8d2-4958ba502338" path="/var/lib/kubelet/pods/5432881f-c368-4d90-b8d2-4958ba502338/volumes" Nov 01 15:48:40.864400 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:48:40.864364 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:48:43.946701 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:43.946647 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" podUID="3eab1985-6883-4db6-ac14-0d3ae3159c20" containerName="registry" containerID="cri-o://cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de" gracePeriod=30 Nov 01 15:48:44.201514 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.201450 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:48:44.282828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.282792 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.282828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.282835 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.283074 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.282876 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.283074 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.282903 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.283074 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.282958 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.283074 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.282985 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.283074 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.283018 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.283321 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.283090 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\") pod \"3eab1985-6883-4db6-ac14-0d3ae3159c20\" (UID: \"3eab1985-6883-4db6-ac14-0d3ae3159c20\") " Nov 01 15:48:44.283514 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.283460 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:44.283514 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.283498 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:44.285463 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.285411 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:44.285586 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.285501 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:44.285789 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.285764 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:44.285857 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.285771 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:44.285895 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.285867 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "kube-api-access-6rj4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:44.289902 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.289852 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted" pod "3eab1985-6883-4db6-ac14-0d3ae3159c20" (UID: "3eab1985-6883-4db6-ac14-0d3ae3159c20"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:48:44.327112 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.327082 4911 generic.go:356] "Generic (PLEG): container finished" podID="3eab1985-6883-4db6-ac14-0d3ae3159c20" containerID="cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de" exitCode=0 Nov 01 15:48:44.327266 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.327131 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" Nov 01 15:48:44.327266 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.327165 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" event={"ID":"3eab1985-6883-4db6-ac14-0d3ae3159c20","Type":"ContainerDied","Data":"cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de"} Nov 01 15:48:44.327266 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.327215 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84cf9dbdbb-c9jvz" event={"ID":"3eab1985-6883-4db6-ac14-0d3ae3159c20","Type":"ContainerDied","Data":"c19b28c4d6d545663a035c94f2e197faa10ad76647572fa396dc49fcec441112"} Nov 01 15:48:44.327266 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.327234 4911 scope.go:122] "RemoveContainer" containerID="cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de" Nov 01 15:48:44.335361 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.335346 4911 scope.go:122] "RemoveContainer" containerID="cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de" Nov 01 15:48:44.335606 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:48:44.335590 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de\": container with ID starting with cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de not found: ID does not exist" containerID="cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de" Nov 01 15:48:44.335657 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.335619 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de"} err="failed to get container status \"cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de\": rpc error: code = NotFound desc = could not find container \"cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de\": container with ID starting with cb9481f2dbcc86e199d3b5442d1c01689c7cc48fe7d45624a8b718de0d93f0de not found: ID does not exist" Nov 01 15:48:44.347261 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.347240 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84cf9dbdbb-c9jvz"] Nov 01 15:48:44.354126 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.354105 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84cf9dbdbb-c9jvz"] Nov 01 15:48:44.384655 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384632 4911 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3eab1985-6883-4db6-ac14-0d3ae3159c20-ca-trust-extracted\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:44.384655 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384655 4911 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-bound-sa-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:44.384828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384666 4911 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:44.384828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384675 4911 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-trusted-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:44.384828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384684 4911 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-image-registry-private-configuration\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:44.384828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384693 4911 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3eab1985-6883-4db6-ac14-0d3ae3159c20-registry-certificates\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:44.384828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384702 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rj4g\" (UniqueName: \"kubernetes.io/projected/3eab1985-6883-4db6-ac14-0d3ae3159c20-kube-api-access-6rj4g\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:44.384828 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:44.384711 4911 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3eab1985-6883-4db6-ac14-0d3ae3159c20-installation-pull-secrets\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:45.967224 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:45.967194 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eab1985-6883-4db6-ac14-0d3ae3159c20" path="/var/lib/kubelet/pods/3eab1985-6883-4db6-ac14-0d3ae3159c20/volumes" Nov 01 15:48:50.350690 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.350655 4911 generic.go:356] "Generic (PLEG): container finished" podID="23a31617-3eb9-499e-a192-f4d24a34e835" containerID="650587bc6b5d8964e5286a66e4a610872a463f6d2fa2e98aeb4111a558dcfe10" exitCode=0 Nov 01 15:48:50.351262 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.350731 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz7xr" event={"ID":"23a31617-3eb9-499e-a192-f4d24a34e835","Type":"ContainerDied","Data":"650587bc6b5d8964e5286a66e4a610872a463f6d2fa2e98aeb4111a558dcfe10"} Nov 01 15:48:50.351262 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.351143 4911 scope.go:122] "RemoveContainer" containerID="650587bc6b5d8964e5286a66e4a610872a463f6d2fa2e98aeb4111a558dcfe10" Nov 01 15:48:50.806543 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.806506 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dmh7c-gpfqf"] Nov 01 15:48:50.806818 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.806807 4911 cpu_manager.go:414] "containerMap: removing container" podUID="5432881f-c368-4d90-b8d2-4958ba502338" containerName="registry" Nov 01 15:48:50.806861 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.806822 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5432881f-c368-4d90-b8d2-4958ba502338" containerName="registry" Nov 01 15:48:50.806861 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.806847 4911 cpu_manager.go:414] "containerMap: removing container" podUID="3eab1985-6883-4db6-ac14-0d3ae3159c20" containerName="registry" Nov 01 15:48:50.806861 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.806852 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eab1985-6883-4db6-ac14-0d3ae3159c20" containerName="registry" Nov 01 15:48:50.806953 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.806898 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="5432881f-c368-4d90-b8d2-4958ba502338" containerName="registry" Nov 01 15:48:50.806953 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.806909 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eab1985-6883-4db6-ac14-0d3ae3159c20" containerName="registry" Nov 01 15:48:50.809887 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.809867 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:50.818953 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.818897 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jbzz7\"" Nov 01 15:48:50.822145 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.822120 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dmh7c-gpfqf"] Nov 01 15:48:50.939049 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.939009 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-service-ca-bundle\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:50.939212 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.939099 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-trusted-ca-bundle\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:50.939212 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.939164 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b478dcb-23eb-4747-8a96-05d1575acbb5-archives-path\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:50.939212 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:50.939189 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdw6\" (UniqueName: \"kubernetes.io/projected/5b478dcb-23eb-4747-8a96-05d1575acbb5-kube-api-access-bqdw6\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.039762 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.039723 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-service-ca-bundle\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.039762 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.039765 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-trusted-ca-bundle\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.039972 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.039796 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b478dcb-23eb-4747-8a96-05d1575acbb5-archives-path\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.039972 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.039813 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdw6\" (UniqueName: \"kubernetes.io/projected/5b478dcb-23eb-4747-8a96-05d1575acbb5-kube-api-access-bqdw6\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.040267 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.040246 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b478dcb-23eb-4747-8a96-05d1575acbb5-archives-path\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.040602 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.040577 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-service-ca-bundle\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.040691 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.040674 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-trusted-ca-bundle\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.048174 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.048149 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdw6\" (UniqueName: \"kubernetes.io/projected/5b478dcb-23eb-4747-8a96-05d1575acbb5-kube-api-access-bqdw6\") pod \"periodic-gathering-dmh7c-gpfqf\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.123857 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.123774 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:48:51.356016 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.355983 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz7xr" event={"ID":"23a31617-3eb9-499e-a192-f4d24a34e835","Type":"ContainerStarted","Data":"5e3cf3943415bff50d9fa3fed59e6570c30589fac5642dbf4e29b0fc24408cf5"} Nov 01 15:48:51.393687 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:51.393608 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dmh7c-gpfqf"] Nov 01 15:48:51.399235 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:48:51.399205 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b478dcb_23eb_4747_8a96_05d1575acbb5.slice/crio-4d0a278de717cf50a94c45bf37b1f20a571eab31bdbf7c4b77d9cf8d9b594fc7 WatchSource:0}: Error finding container 4d0a278de717cf50a94c45bf37b1f20a571eab31bdbf7c4b77d9cf8d9b594fc7: Status 404 returned error can't find the container with id 4d0a278de717cf50a94c45bf37b1f20a571eab31bdbf7c4b77d9cf8d9b594fc7 Nov 01 15:48:52.363975 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:52.363935 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" event={"ID":"5b478dcb-23eb-4747-8a96-05d1575acbb5","Type":"ContainerStarted","Data":"4a73e1940369047ac3ac293dc4fcadb8bedabc6f7939a4197509dbc48417e6bf"} Nov 01 15:48:52.364382 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:52.363999 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" event={"ID":"5b478dcb-23eb-4747-8a96-05d1575acbb5","Type":"ContainerStarted","Data":"4d0a278de717cf50a94c45bf37b1f20a571eab31bdbf7c4b77d9cf8d9b594fc7"} Nov 01 15:48:52.381432 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:52.381388 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" podStartSLOduration=2.381375364 podStartE2EDuration="2.381375364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:52.379419726 +0000 UTC m=+72.523874366" watchObservedRunningTime="2026-11-01 15:48:52.381375364 +0000 UTC m=+72.525830007" Nov 01 15:48:55.377414 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:55.377377 4911 generic.go:356] "Generic (PLEG): container finished" podID="a9662ce2-6f58-4385-8f20-f9ed018fea61" containerID="96dc33b92346765a29b92d9c11d93d7a2442495ab7398c53ac0f9700f7697002" exitCode=0 Nov 01 15:48:55.377887 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:55.377458 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" event={"ID":"a9662ce2-6f58-4385-8f20-f9ed018fea61","Type":"ContainerDied","Data":"96dc33b92346765a29b92d9c11d93d7a2442495ab7398c53ac0f9700f7697002"} Nov 01 15:48:55.377965 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:55.377886 4911 scope.go:122] "RemoveContainer" containerID="96dc33b92346765a29b92d9c11d93d7a2442495ab7398c53ac0f9700f7697002" Nov 01 15:48:56.383307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:48:56.383269 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wwww7" event={"ID":"a9662ce2-6f58-4385-8f20-f9ed018fea61","Type":"ContainerStarted","Data":"ddad208bd6eeab2e2d9de725e1e637586cc5d2ad589b23cb3de2b5b52f1d206b"} Nov 01 15:49:10.873075 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:49:10.873017 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:49:40.863869 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:49:40.863828 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:50:10.864129 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:50:10.864092 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:50:15.682450 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:15.682414 4911 generic.go:356] "Generic (PLEG): container finished" podID="5b478dcb-23eb-4747-8a96-05d1575acbb5" containerID="4a73e1940369047ac3ac293dc4fcadb8bedabc6f7939a4197509dbc48417e6bf" exitCode=0 Nov 01 15:50:15.682833 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:15.682487 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" event={"ID":"5b478dcb-23eb-4747-8a96-05d1575acbb5","Type":"ContainerDied","Data":"4a73e1940369047ac3ac293dc4fcadb8bedabc6f7939a4197509dbc48417e6bf"} Nov 01 15:50:16.804607 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.804568 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:50:16.901545 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.901499 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b478dcb-23eb-4747-8a96-05d1575acbb5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b478dcb-23eb-4747-8a96-05d1575acbb5-archives-path\") pod \"5b478dcb-23eb-4747-8a96-05d1575acbb5\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " Nov 01 15:50:16.901720 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.901585 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-service-ca-bundle\") pod \"5b478dcb-23eb-4747-8a96-05d1575acbb5\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " Nov 01 15:50:16.901720 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.901614 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b478dcb-23eb-4747-8a96-05d1575acbb5-kube-api-access-bqdw6\" (UniqueName: \"kubernetes.io/projected/5b478dcb-23eb-4747-8a96-05d1575acbb5-kube-api-access-bqdw6\") pod \"5b478dcb-23eb-4747-8a96-05d1575acbb5\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " Nov 01 15:50:16.901720 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.901638 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-trusted-ca-bundle\") pod \"5b478dcb-23eb-4747-8a96-05d1575acbb5\" (UID: \"5b478dcb-23eb-4747-8a96-05d1575acbb5\") " Nov 01 15:50:16.901973 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.901881 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b478dcb-23eb-4747-8a96-05d1575acbb5-archives-path" pod "5b478dcb-23eb-4747-8a96-05d1575acbb5" (UID: "5b478dcb-23eb-4747-8a96-05d1575acbb5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:50:16.902010 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.901957 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-service-ca-bundle" pod "5b478dcb-23eb-4747-8a96-05d1575acbb5" (UID: "5b478dcb-23eb-4747-8a96-05d1575acbb5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:50:16.902064 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.902025 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-trusted-ca-bundle" pod "5b478dcb-23eb-4747-8a96-05d1575acbb5" (UID: "5b478dcb-23eb-4747-8a96-05d1575acbb5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:50:16.903811 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:16.903790 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b478dcb-23eb-4747-8a96-05d1575acbb5-kube-api-access-bqdw6" pod "5b478dcb-23eb-4747-8a96-05d1575acbb5" (UID: "5b478dcb-23eb-4747-8a96-05d1575acbb5"). InnerVolumeSpecName "kube-api-access-bqdw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:50:17.002514 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:17.002475 4911 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5b478dcb-23eb-4747-8a96-05d1575acbb5-archives-path\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:17.002514 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:17.002514 4911 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-service-ca-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:17.002722 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:17.002524 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqdw6\" (UniqueName: \"kubernetes.io/projected/5b478dcb-23eb-4747-8a96-05d1575acbb5-kube-api-access-bqdw6\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:17.002722 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:17.002535 4911 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b478dcb-23eb-4747-8a96-05d1575acbb5-trusted-ca-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:17.691010 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:17.690982 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" Nov 01 15:50:17.691188 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:17.690979 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dmh7c-gpfqf" event={"ID":"5b478dcb-23eb-4747-8a96-05d1575acbb5","Type":"ContainerDied","Data":"4d0a278de717cf50a94c45bf37b1f20a571eab31bdbf7c4b77d9cf8d9b594fc7"} Nov 01 15:50:17.691188 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:50:17.691111 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d0a278de717cf50a94c45bf37b1f20a571eab31bdbf7c4b77d9cf8d9b594fc7" Nov 01 15:50:40.865460 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:50:40.865411 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:51:10.863970 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:51:10.863931 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:51:40.863905 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:51:40.863862 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:52:10.864140 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:52:10.864102 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:52:26.416374 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.416336 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9"] Nov 01 15:52:26.416756 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.416660 4911 cpu_manager.go:414] "containerMap: removing container" podUID="5b478dcb-23eb-4747-8a96-05d1575acbb5" containerName="insights-gathering" Nov 01 15:52:26.416756 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.416672 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b478dcb-23eb-4747-8a96-05d1575acbb5" containerName="insights-gathering" Nov 01 15:52:26.416756 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.416740 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b478dcb-23eb-4747-8a96-05d1575acbb5" containerName="insights-gathering" Nov 01 15:52:26.419737 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.419713 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.425408 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.425380 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 15:52:26.426275 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.426256 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 15:52:26.426367 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.426336 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-x6pqc\"" Nov 01 15:52:26.434590 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.434567 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9"] Nov 01 15:52:26.451980 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.451949 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.452148 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.452009 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclpn\" (UniqueName: \"kubernetes.io/projected/e0df6725-a199-496f-8160-0867c9b169e0-kube-api-access-gclpn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.452148 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.452125 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.548794 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.548755 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t"] Nov 01 15:52:26.552295 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.552271 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.552533 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.552508 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.552601 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.552560 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.552664 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.552647 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gclpn\" (UniqueName: \"kubernetes.io/projected/e0df6725-a199-496f-8160-0867c9b169e0-kube-api-access-gclpn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.552942 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.552918 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.553017 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.552996 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.556131 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.556099 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-trnjw\"" Nov 01 15:52:26.561712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.561689 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t"] Nov 01 15:52:26.564050 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.564011 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclpn\" (UniqueName: \"kubernetes.io/projected/e0df6725-a199-496f-8160-0867c9b169e0-kube-api-access-gclpn\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.653196 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.653161 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.653365 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.653224 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.653365 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.653303 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctw4\" (UniqueName: \"kubernetes.io/projected/f2f7881c-54c2-4864-bb13-a8aafc6d0478-kube-api-access-tctw4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.735818 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.735792 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:26.754146 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.754116 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.754261 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.754186 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.754261 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.754207 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctw4\" (UniqueName: \"kubernetes.io/projected/f2f7881c-54c2-4864-bb13-a8aafc6d0478-kube-api-access-tctw4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.754534 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.754505 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.754656 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.754550 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.765366 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.765340 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctw4\" (UniqueName: \"kubernetes.io/projected/f2f7881c-54c2-4864-bb13-a8aafc6d0478-kube-api-access-tctw4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:26.870666 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:26.870611 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:27.008297 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:27.008271 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9"] Nov 01 15:52:27.010364 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:27.010340 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0df6725_a199_496f_8160_0867c9b169e0.slice/crio-fc55815f44b93d1e309a37aed6c136b81abc34ab6546c10f3bbf8ef867f40c5d WatchSource:0}: Error finding container fc55815f44b93d1e309a37aed6c136b81abc34ab6546c10f3bbf8ef867f40c5d: Status 404 returned error can't find the container with id fc55815f44b93d1e309a37aed6c136b81abc34ab6546c10f3bbf8ef867f40c5d Nov 01 15:52:27.144193 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:27.144163 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t"] Nov 01 15:52:27.147195 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:27.147168 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f7881c_54c2_4864_bb13_a8aafc6d0478.slice/crio-3a8aeef327a0a747a42bcf4fb4ecbf07f8e8643935c3beb3a4fabcec3f17e8c4 WatchSource:0}: Error finding container 3a8aeef327a0a747a42bcf4fb4ecbf07f8e8643935c3beb3a4fabcec3f17e8c4: Status 404 returned error can't find the container with id 3a8aeef327a0a747a42bcf4fb4ecbf07f8e8643935c3beb3a4fabcec3f17e8c4 Nov 01 15:52:27.151520 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:27.151494 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" event={"ID":"e0df6725-a199-496f-8160-0867c9b169e0","Type":"ContainerStarted","Data":"fc55815f44b93d1e309a37aed6c136b81abc34ab6546c10f3bbf8ef867f40c5d"} Nov 01 15:52:28.158053 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.157983 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" event={"ID":"f2f7881c-54c2-4864-bb13-a8aafc6d0478","Type":"ContainerStarted","Data":"3a8aeef327a0a747a42bcf4fb4ecbf07f8e8643935c3beb3a4fabcec3f17e8c4"} Nov 01 15:52:28.857803 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.857765 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg"] Nov 01 15:52:28.861550 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.861525 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:28.867092 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.867067 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-5hjvz\"" Nov 01 15:52:28.870258 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.870231 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg"] Nov 01 15:52:28.971188 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.971148 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:28.971389 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.971302 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:28.971389 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:28.971365 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2w7\" (UniqueName: \"kubernetes.io/projected/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-kube-api-access-gw2w7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.072510 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.072473 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.072700 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.072533 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2w7\" (UniqueName: \"kubernetes.io/projected/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-kube-api-access-gw2w7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.072700 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.072581 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.072928 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.072905 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.073008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.072959 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.084034 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.083999 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2w7\" (UniqueName: \"kubernetes.io/projected/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-kube-api-access-gw2w7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.181191 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.181115 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:29.471132 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:29.470899 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg"] Nov 01 15:52:30.075013 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.073135 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx"] Nov 01 15:52:30.077305 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.077280 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.090241 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.088084 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9jpsx\"" Nov 01 15:52:30.090241 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.089910 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx"] Nov 01 15:52:30.170168 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.170093 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" event={"ID":"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4","Type":"ContainerStarted","Data":"a163e5b09110bf8eca41e19b7fa7195710775ad80fec7f53bbee4e07106b2d09"} Nov 01 15:52:30.177548 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.176244 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz"] Nov 01 15:52:30.181082 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.180355 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.181435 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.181403 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jkv\" (UniqueName: \"kubernetes.io/projected/797b36d4-cdf5-4e17-a206-871d07ad0915-kube-api-access-v8jkv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.181815 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.181465 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.181815 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.181505 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.187215 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.187180 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-jnnbx\"" Nov 01 15:52:30.189758 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.189660 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz"] Nov 01 15:52:30.282318 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282279 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.282491 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282326 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.282491 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282415 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jkv\" (UniqueName: \"kubernetes.io/projected/797b36d4-cdf5-4e17-a206-871d07ad0915-kube-api-access-v8jkv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.282491 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282457 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzwd\" (UniqueName: \"kubernetes.io/projected/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-kube-api-access-8bzwd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.282662 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282502 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.282662 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282548 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.282983 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282960 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.283072 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.282983 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.284025 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.283992 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv"] Nov 01 15:52:30.288563 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.288542 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.293950 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.293929 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-hth6g\"" Nov 01 15:52:30.294442 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.294416 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jkv\" (UniqueName: \"kubernetes.io/projected/797b36d4-cdf5-4e17-a206-871d07ad0915-kube-api-access-v8jkv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.296739 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.296712 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv"] Nov 01 15:52:30.378236 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.378155 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89"] Nov 01 15:52:30.382244 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.382215 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.382984 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.382958 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.383144 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.383006 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.383144 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.383110 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.383333 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.383150 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.383333 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.383202 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9cc6\" (UniqueName: \"kubernetes.io/projected/4e15b807-cb0c-49b1-a984-f121464d9273-kube-api-access-m9cc6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.383333 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.383259 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzwd\" (UniqueName: \"kubernetes.io/projected/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-kube-api-access-8bzwd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.383539 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.383461 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.383782 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.383758 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.390032 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.390005 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89"] Nov 01 15:52:30.392880 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.392857 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pxb2m\"" Nov 01 15:52:30.392983 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.392945 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzwd\" (UniqueName: \"kubernetes.io/projected/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-kube-api-access-8bzwd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.402065 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.401888 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:30.484740 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.484693 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.484901 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.484794 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.484901 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.484840 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qzs\" (UniqueName: \"kubernetes.io/projected/1962f9ad-1dfc-4e1e-a470-694a608a9082-kube-api-access-k7qzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.484901 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.484871 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.485783 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.484911 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9cc6\" (UniqueName: \"kubernetes.io/projected/4e15b807-cb0c-49b1-a984-f121464d9273-kube-api-access-m9cc6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.485783 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.484955 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.485783 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.485436 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.485783 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.485447 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.497021 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.496952 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9cc6\" (UniqueName: \"kubernetes.io/projected/4e15b807-cb0c-49b1-a984-f121464d9273-kube-api-access-m9cc6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.502886 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.502679 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:30.586296 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.586128 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qzs\" (UniqueName: \"kubernetes.io/projected/1962f9ad-1dfc-4e1e-a470-694a608a9082-kube-api-access-k7qzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.586296 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.586194 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.586296 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.586217 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.586600 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.586581 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.587595 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.587060 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.601704 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.601446 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qzs\" (UniqueName: \"kubernetes.io/projected/1962f9ad-1dfc-4e1e-a470-694a608a9082-kube-api-access-k7qzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.620312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.620112 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:30.691807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.691773 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx"] Nov 01 15:52:30.696178 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:30.696144 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797b36d4_cdf5_4e17_a206_871d07ad0915.slice/crio-90bd1078b777cc754dfae00c392cd2f7126fef1be307c9615de39f7b4a1ca2f3 WatchSource:0}: Error finding container 90bd1078b777cc754dfae00c392cd2f7126fef1be307c9615de39f7b4a1ca2f3: Status 404 returned error can't find the container with id 90bd1078b777cc754dfae00c392cd2f7126fef1be307c9615de39f7b4a1ca2f3 Nov 01 15:52:30.713709 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.713511 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:30.807817 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.807780 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz"] Nov 01 15:52:30.812003 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:30.811972 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaeeb051_99b4_4ae0_b0c1_37bcd23264d6.slice/crio-64100bdf88aa80c1b97012a316d57e94361bf146b60de9bd45ecafa9ce3e7e4c WatchSource:0}: Error finding container 64100bdf88aa80c1b97012a316d57e94361bf146b60de9bd45ecafa9ce3e7e4c: Status 404 returned error can't find the container with id 64100bdf88aa80c1b97012a316d57e94361bf146b60de9bd45ecafa9ce3e7e4c Nov 01 15:52:30.877946 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.877872 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l"] Nov 01 15:52:30.882735 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.882674 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.887754 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.887729 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 15:52:30.889087 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.889028 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tkjjf\"" Nov 01 15:52:30.896102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.894136 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l"] Nov 01 15:52:30.921632 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.921608 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv"] Nov 01 15:52:30.925696 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:30.925629 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e15b807_cb0c_49b1_a984_f121464d9273.slice/crio-1ba074c0e8912595ab448fe998e1c903b8ff8ed467ce72443b83d1f77198a6e0 WatchSource:0}: Error finding container 1ba074c0e8912595ab448fe998e1c903b8ff8ed467ce72443b83d1f77198a6e0: Status 404 returned error can't find the container with id 1ba074c0e8912595ab448fe998e1c903b8ff8ed467ce72443b83d1f77198a6e0 Nov 01 15:52:30.955487 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.955457 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll"] Nov 01 15:52:30.961020 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.960995 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.971892 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.971724 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll"] Nov 01 15:52:30.986599 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.986556 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-f9tcm\"" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991297 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991328 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991347 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991363 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d302b57f-5184-48fd-b8e3-0e22166dd567-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991388 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991411 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbnq\" (UniqueName: \"kubernetes.io/projected/d302b57f-5184-48fd-b8e3-0e22166dd567-kube-api-access-fmbnq\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991433 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991466 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991485 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2sm\" (UniqueName: \"kubernetes.io/projected/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-kube-api-access-th2sm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991626 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991665 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991691 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991720 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991746 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991773 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991800 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991834 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:30.992008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:30.991858 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.031232 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.029906 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89"] Nov 01 15:52:31.033297 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:31.033258 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1962f9ad_1dfc_4e1e_a470_694a608a9082.slice/crio-498c763e54df8d6a321d26aeb4072df357b2c884fe4f62a47fcd3cee1d44144c WatchSource:0}: Error finding container 498c763e54df8d6a321d26aeb4072df357b2c884fe4f62a47fcd3cee1d44144c: Status 404 returned error can't find the container with id 498c763e54df8d6a321d26aeb4072df357b2c884fe4f62a47fcd3cee1d44144c Nov 01 15:52:31.092633 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092572 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.092807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092640 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.092807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092670 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2sm\" (UniqueName: \"kubernetes.io/projected/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-kube-api-access-th2sm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.092807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092698 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.092807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092736 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.092807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092763 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.092807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092790 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092816 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092843 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092871 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092906 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092929 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092967 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.092998 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.093033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.093029 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.093377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.093076 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d302b57f-5184-48fd-b8e3-0e22166dd567-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.093377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.093108 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.093377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.093148 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbnq\" (UniqueName: \"kubernetes.io/projected/d302b57f-5184-48fd-b8e3-0e22166dd567-kube-api-access-fmbnq\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.093868 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.093841 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.094203 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.094184 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.095358 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.095410 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.095630 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.095708 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.096114 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.096363 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.096583 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.096624 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d302b57f-5184-48fd-b8e3-0e22166dd567-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.096843 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.096731 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.099279 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.099249 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.099521 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.099502 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.099781 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.099720 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.102214 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.102166 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbnq\" (UniqueName: \"kubernetes.io/projected/d302b57f-5184-48fd-b8e3-0e22166dd567-kube-api-access-fmbnq\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.103313 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.103292 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2sm\" (UniqueName: \"kubernetes.io/projected/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-kube-api-access-th2sm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.104048 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.104013 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37e3bdc-1a02-41cb-8c76-61d11cbd86b8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-9rb5l\" (UID: \"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.105569 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.105533 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d302b57f-5184-48fd-b8e3-0e22166dd567-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll\" (UID: \"d302b57f-5184-48fd-b8e3-0e22166dd567\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.178271 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.178128 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" event={"ID":"4e15b807-cb0c-49b1-a984-f121464d9273","Type":"ContainerStarted","Data":"1ba074c0e8912595ab448fe998e1c903b8ff8ed467ce72443b83d1f77198a6e0"} Nov 01 15:52:31.180404 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.180370 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" event={"ID":"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6","Type":"ContainerStarted","Data":"64100bdf88aa80c1b97012a316d57e94361bf146b60de9bd45ecafa9ce3e7e4c"} Nov 01 15:52:31.183201 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.183168 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" event={"ID":"1962f9ad-1dfc-4e1e-a470-694a608a9082","Type":"ContainerStarted","Data":"498c763e54df8d6a321d26aeb4072df357b2c884fe4f62a47fcd3cee1d44144c"} Nov 01 15:52:31.185555 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.185481 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" event={"ID":"797b36d4-cdf5-4e17-a206-871d07ad0915","Type":"ContainerStarted","Data":"90bd1078b777cc754dfae00c392cd2f7126fef1be307c9615de39f7b4a1ca2f3"} Nov 01 15:52:31.213900 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.213866 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:31.282733 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.282440 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:31.516005 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.515947 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l"] Nov 01 15:52:31.522636 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:31.522602 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37e3bdc_1a02_41cb_8c76_61d11cbd86b8.slice/crio-e791715fd34eb9d0c825a9eb0fc4cacbf208c793b3b95d00b74cad5450891745 WatchSource:0}: Error finding container e791715fd34eb9d0c825a9eb0fc4cacbf208c793b3b95d00b74cad5450891745: Status 404 returned error can't find the container with id e791715fd34eb9d0c825a9eb0fc4cacbf208c793b3b95d00b74cad5450891745 Nov 01 15:52:31.612229 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:31.612193 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll"] Nov 01 15:52:32.194374 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:32.194280 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" event={"ID":"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8","Type":"ContainerStarted","Data":"e791715fd34eb9d0c825a9eb0fc4cacbf208c793b3b95d00b74cad5450891745"} Nov 01 15:52:32.197136 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:32.197067 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" event={"ID":"d302b57f-5184-48fd-b8e3-0e22166dd567","Type":"ContainerStarted","Data":"75ed4e398334f3a09a685154be575b61110eba542ecb9c7c00ca1ac00a4c00ce"} Nov 01 15:52:39.969218 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:39.968663 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:39.969218 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:39.968755 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:39.969218 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:39.968796 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:40.016897 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.016875 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 15:52:40.019169 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.019148 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 15:52:40.236377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.235482 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:40.236377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.235568 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:40.236377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.235608 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:52:40.274166 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.273002 4911 generic.go:356] "Generic (PLEG): container finished" podID="e0df6725-a199-496f-8160-0867c9b169e0" containerID="beef1d04c02e1188634ad849ca4c6baf3230b78db240c89562093afc19ec3b32" exitCode=0 Nov 01 15:52:40.274166 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.273122 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" event={"ID":"e0df6725-a199-496f-8160-0867c9b169e0","Type":"ContainerDied","Data":"beef1d04c02e1188634ad849ca4c6baf3230b78db240c89562093afc19ec3b32"} Nov 01 15:52:40.280354 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.280321 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" event={"ID":"d302b57f-5184-48fd-b8e3-0e22166dd567","Type":"ContainerStarted","Data":"d0c957b04b066b77f4e3996299928bca16167ef3b0f11a09d41a42e446965e33"} Nov 01 15:52:40.282779 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.282758 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:40.283643 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.283618 4911 generic.go:356] "Generic (PLEG): container finished" podID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerID="49210ff3b0bd68896b4bdac82f3f63a3fd8d2bbc70f9b1e4f11b84114e8f9722" exitCode=0 Nov 01 15:52:40.283743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.283693 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" event={"ID":"f2f7881c-54c2-4864-bb13-a8aafc6d0478","Type":"ContainerDied","Data":"49210ff3b0bd68896b4bdac82f3f63a3fd8d2bbc70f9b1e4f11b84114e8f9722"} Nov 01 15:52:40.331982 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:40.331925 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" podStartSLOduration=1.984744413 podStartE2EDuration="10.331905014s" totalImagesPullingTime="8.347160601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:40.330287083 +0000 UTC m=+300.474741725" watchObservedRunningTime="2026-11-01 15:52:40.331905014 +0000 UTC m=+300.476359657" Nov 01 15:52:40.875594 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:52:40.875559 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:52:41.289265 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.289235 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:41.291176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.291147 4911 generic.go:356] "Generic (PLEG): container finished" podID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerID="cf000ca91705f0611530eb8f8b3bfaf0464a0ce60846e7182eb08e6b297f5c9c" exitCode=0 Nov 01 15:52:41.291302 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.291236 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" event={"ID":"f2f7881c-54c2-4864-bb13-a8aafc6d0478","Type":"ContainerDied","Data":"cf000ca91705f0611530eb8f8b3bfaf0464a0ce60846e7182eb08e6b297f5c9c"} Nov 01 15:52:41.294007 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.293405 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" event={"ID":"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8","Type":"ContainerStarted","Data":"865f013190fc087ac222d9a18d64d7dc11c9f641147e95c7a9e240fedb8c80ba"} Nov 01 15:52:41.295530 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.295507 4911 generic.go:356] "Generic (PLEG): container finished" podID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerID="cf474a37ac3ff977a34a87f86b56ecbd5be4e2d441468646769a2d8df889b81f" exitCode=0 Nov 01 15:52:41.295619 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.295576 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" event={"ID":"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6","Type":"ContainerDied","Data":"cf474a37ac3ff977a34a87f86b56ecbd5be4e2d441468646769a2d8df889b81f"} Nov 01 15:52:41.297496 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.297399 4911 generic.go:356] "Generic (PLEG): container finished" podID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerID="668be23b0dca14c254f100642871bee7ceaa8152188e2547f55b802bdacf23c0" exitCode=0 Nov 01 15:52:41.297719 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.297642 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" event={"ID":"1962f9ad-1dfc-4e1e-a470-694a608a9082","Type":"ContainerDied","Data":"668be23b0dca14c254f100642871bee7ceaa8152188e2547f55b802bdacf23c0"} Nov 01 15:52:41.300215 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.300193 4911 generic.go:356] "Generic (PLEG): container finished" podID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerID="cec9e744bc8b63b9a86c60d6666977038a6c19ddda8f45aff5fb15ab47c8381b" exitCode=0 Nov 01 15:52:41.300332 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.300297 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" event={"ID":"797b36d4-cdf5-4e17-a206-871d07ad0915","Type":"ContainerDied","Data":"cec9e744bc8b63b9a86c60d6666977038a6c19ddda8f45aff5fb15ab47c8381b"} Nov 01 15:52:41.302305 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.302279 4911 generic.go:356] "Generic (PLEG): container finished" podID="4e15b807-cb0c-49b1-a984-f121464d9273" containerID="4c19dbc1df5bd572d855141b4e5a7fcf5b7416fdedf984973f3b4c36b0f2cf2b" exitCode=0 Nov 01 15:52:41.303320 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.303298 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" event={"ID":"4e15b807-cb0c-49b1-a984-f121464d9273","Type":"ContainerDied","Data":"4c19dbc1df5bd572d855141b4e5a7fcf5b7416fdedf984973f3b4c36b0f2cf2b"} Nov 01 15:52:41.303424 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.303347 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:41.304146 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.304123 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll" Nov 01 15:52:41.418552 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:41.418475 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podStartSLOduration=2.70987905 podStartE2EDuration="11.418461755s" totalImagesPullingTime="8.708582705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:41.41726387 +0000 UTC m=+301.561718523" watchObservedRunningTime="2026-11-01 15:52:41.418461755 +0000 UTC m=+301.562916396" Nov 01 15:52:42.215395 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:42.215350 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:42.221803 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:42.221768 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:42.309253 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:42.309113 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:42.310679 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:42.310656 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:52:43.315010 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.314976 4911 generic.go:356] "Generic (PLEG): container finished" podID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerID="29407c31c46ed5fffc32748861b0aef960548ecfa45d8dc14e2ae0d1778205a2" exitCode=0 Nov 01 15:52:43.315528 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.315073 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" event={"ID":"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6","Type":"ContainerDied","Data":"29407c31c46ed5fffc32748861b0aef960548ecfa45d8dc14e2ae0d1778205a2"} Nov 01 15:52:43.316544 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.316521 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:52:43.317380 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.317353 4911 generic.go:356] "Generic (PLEG): container finished" podID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerID="dbbcefe720c148d451dbc7dba6fcfea72467d7b5e945b0dea18418aa7b3df17f" exitCode=0 Nov 01 15:52:43.317556 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.317440 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" event={"ID":"797b36d4-cdf5-4e17-a206-871d07ad0915","Type":"ContainerDied","Data":"dbbcefe720c148d451dbc7dba6fcfea72467d7b5e945b0dea18418aa7b3df17f"} Nov 01 15:52:43.320092 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.319996 4911 generic.go:356] "Generic (PLEG): container finished" podID="4e15b807-cb0c-49b1-a984-f121464d9273" containerID="c3dc3cc97c85b63b5e4fc4e729b3552ce2cdfe6200acb35abe7df2cbeb679a29" exitCode=0 Nov 01 15:52:43.320159 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.320143 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" event={"ID":"4e15b807-cb0c-49b1-a984-f121464d9273","Type":"ContainerDied","Data":"c3dc3cc97c85b63b5e4fc4e729b3552ce2cdfe6200acb35abe7df2cbeb679a29"} Nov 01 15:52:43.321774 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.321752 4911 generic.go:356] "Generic (PLEG): container finished" podID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerID="c84713aaecd1b48862fea383c158031b00383bb299c04e3cc096d3883953c002" exitCode=0 Nov 01 15:52:43.321865 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:43.321791 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" event={"ID":"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4","Type":"ContainerDied","Data":"c84713aaecd1b48862fea383c158031b00383bb299c04e3cc096d3883953c002"} Nov 01 15:52:47.359951 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:47.358946 4911 generic.go:356] "Generic (PLEG): container finished" podID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerID="9c3d19dd36f3678c428e5118eebf488ee256b721c446a5ed0e11fbe34d0d5240" exitCode=0 Nov 01 15:52:47.359951 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:47.359076 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" event={"ID":"1962f9ad-1dfc-4e1e-a470-694a608a9082","Type":"ContainerDied","Data":"9c3d19dd36f3678c428e5118eebf488ee256b721c446a5ed0e11fbe34d0d5240"} Nov 01 15:52:49.377542 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:49.376526 4911 generic.go:356] "Generic (PLEG): container finished" podID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerID="4944b3d3505b0e225bf113500dd18ae7236a011c40ece7a452cc296488359fe7" exitCode=0 Nov 01 15:52:49.377542 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:49.376676 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" event={"ID":"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4","Type":"ContainerDied","Data":"4944b3d3505b0e225bf113500dd18ae7236a011c40ece7a452cc296488359fe7"} Nov 01 15:52:50.386910 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:50.385873 4911 generic.go:356] "Generic (PLEG): container finished" podID="e0df6725-a199-496f-8160-0867c9b169e0" containerID="08bf66d2157c7160c1a7e3779266736d6c21527d7031207c081f91eb7f223c2d" exitCode=0 Nov 01 15:52:50.386910 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:50.385931 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" event={"ID":"e0df6725-a199-496f-8160-0867c9b169e0","Type":"ContainerDied","Data":"08bf66d2157c7160c1a7e3779266736d6c21527d7031207c081f91eb7f223c2d"} Nov 01 15:52:52.406462 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:52.406419 4911 generic.go:356] "Generic (PLEG): container finished" podID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerID="0f761a92aabdd1101453796407ec8b1f3f88ec44c61594ba87a6d1d4c53f3eaa" exitCode=0 Nov 01 15:52:52.407137 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:52.406550 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" event={"ID":"f2f7881c-54c2-4864-bb13-a8aafc6d0478","Type":"ContainerDied","Data":"0f761a92aabdd1101453796407ec8b1f3f88ec44c61594ba87a6d1d4c53f3eaa"} Nov 01 15:52:53.631479 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.631104 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:53.726890 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.725851 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-bundle\") pod \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " Nov 01 15:52:53.726890 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.725994 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-util\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-util\") pod \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " Nov 01 15:52:53.726890 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.726029 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f7881c-54c2-4864-bb13-a8aafc6d0478-kube-api-access-tctw4\" (UniqueName: \"kubernetes.io/projected/f2f7881c-54c2-4864-bb13-a8aafc6d0478-kube-api-access-tctw4\") pod \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\" (UID: \"f2f7881c-54c2-4864-bb13-a8aafc6d0478\") " Nov 01 15:52:53.729701 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.729650 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-bundle" pod "f2f7881c-54c2-4864-bb13-a8aafc6d0478" (UID: "f2f7881c-54c2-4864-bb13-a8aafc6d0478"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:53.730822 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.730740 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f7881c-54c2-4864-bb13-a8aafc6d0478-kube-api-access-tctw4" pod "f2f7881c-54c2-4864-bb13-a8aafc6d0478" (UID: "f2f7881c-54c2-4864-bb13-a8aafc6d0478"). InnerVolumeSpecName "kube-api-access-tctw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:53.734526 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.734497 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-util" pod "f2f7881c-54c2-4864-bb13-a8aafc6d0478" (UID: "f2f7881c-54c2-4864-bb13-a8aafc6d0478"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:53.827143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.827106 4911 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-util\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:53.827143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.827135 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctw4\" (UniqueName: \"kubernetes.io/projected/f2f7881c-54c2-4864-bb13-a8aafc6d0478-kube-api-access-tctw4\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:53.827143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:53.827146 4911 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2f7881c-54c2-4864-bb13-a8aafc6d0478-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:54.439507 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:54.439393 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" event={"ID":"f2f7881c-54c2-4864-bb13-a8aafc6d0478","Type":"ContainerDied","Data":"3a8aeef327a0a747a42bcf4fb4ecbf07f8e8643935c3beb3a4fabcec3f17e8c4"} Nov 01 15:52:54.439709 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:54.439552 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8aeef327a0a747a42bcf4fb4ecbf07f8e8643935c3beb3a4fabcec3f17e8c4" Nov 01 15:52:54.439778 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:54.439721 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7168s6t" Nov 01 15:52:56.460226 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:56.460190 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" event={"ID":"4e15b807-cb0c-49b1-a984-f121464d9273","Type":"ContainerStarted","Data":"173936a7e2cad480b2be95686ff54b93d07374f5a518bac52363489aa3d57a6d"} Nov 01 15:52:56.463325 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:56.463292 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" event={"ID":"1962f9ad-1dfc-4e1e-a470-694a608a9082","Type":"ContainerStarted","Data":"775a11c452be36dfab406bb181d42fd2476e9e77f956dc9b5be6141a92cfcd9e"} Nov 01 15:52:56.481010 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:56.480928 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" podStartSLOduration=2.532132552 podStartE2EDuration="26.48090786s" totalImagesPullingTime="23.948775308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:56.479668599 +0000 UTC m=+316.624123241" watchObservedRunningTime="2026-11-01 15:52:56.48090786 +0000 UTC m=+316.625362530" Nov 01 15:52:57.469488 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.469450 4911 generic.go:356] "Generic (PLEG): container finished" podID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerID="6e785c29c45aa93a49afe8d8c440a528f83949826b94ce884d9809d1d99d1ed8" exitCode=0 Nov 01 15:52:57.469933 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.469534 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" event={"ID":"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4","Type":"ContainerDied","Data":"6e785c29c45aa93a49afe8d8c440a528f83949826b94ce884d9809d1d99d1ed8"} Nov 01 15:52:57.471571 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.471545 4911 generic.go:356] "Generic (PLEG): container finished" podID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerID="5f499e58c2397496ca9c167739bf8b7cec7f0ec2498a579456a13822494fa006" exitCode=0 Nov 01 15:52:57.471714 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.471627 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" event={"ID":"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6","Type":"ContainerDied","Data":"5f499e58c2397496ca9c167739bf8b7cec7f0ec2498a579456a13822494fa006"} Nov 01 15:52:57.473816 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.473792 4911 generic.go:356] "Generic (PLEG): container finished" podID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerID="775a11c452be36dfab406bb181d42fd2476e9e77f956dc9b5be6141a92cfcd9e" exitCode=0 Nov 01 15:52:57.473935 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.473864 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" event={"ID":"1962f9ad-1dfc-4e1e-a470-694a608a9082","Type":"ContainerDied","Data":"775a11c452be36dfab406bb181d42fd2476e9e77f956dc9b5be6141a92cfcd9e"} Nov 01 15:52:57.475721 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.475702 4911 generic.go:356] "Generic (PLEG): container finished" podID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerID="f9d34a922d89b7e2108c3828557772d71e882a8acb0a412cbc7944587f6115d7" exitCode=0 Nov 01 15:52:57.475809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.475784 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" event={"ID":"797b36d4-cdf5-4e17-a206-871d07ad0915","Type":"ContainerDied","Data":"f9d34a922d89b7e2108c3828557772d71e882a8acb0a412cbc7944587f6115d7"} Nov 01 15:52:57.477696 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.477675 4911 generic.go:356] "Generic (PLEG): container finished" podID="4e15b807-cb0c-49b1-a984-f121464d9273" containerID="173936a7e2cad480b2be95686ff54b93d07374f5a518bac52363489aa3d57a6d" exitCode=0 Nov 01 15:52:57.477769 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.477751 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" event={"ID":"4e15b807-cb0c-49b1-a984-f121464d9273","Type":"ContainerDied","Data":"173936a7e2cad480b2be95686ff54b93d07374f5a518bac52363489aa3d57a6d"} Nov 01 15:52:57.487837 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:57.487796 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" podStartSLOduration=3.385297144 podStartE2EDuration="27.487784593s" totalImagesPullingTime="24.102487449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:56.498260834 +0000 UTC m=+316.642715481" watchObservedRunningTime="2026-11-01 15:52:57.487784593 +0000 UTC m=+317.632239233" Nov 01 15:52:58.485669 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.485623 4911 generic.go:356] "Generic (PLEG): container finished" podID="e0df6725-a199-496f-8160-0867c9b169e0" containerID="c3e7bd38a98f367d90e73c8c61c1586ca79d74a6b279ed675f0f35edf72a98e7" exitCode=0 Nov 01 15:52:58.486175 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.485698 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" event={"ID":"e0df6725-a199-496f-8160-0867c9b169e0","Type":"ContainerDied","Data":"c3e7bd38a98f367d90e73c8c61c1586ca79d74a6b279ed675f0f35edf72a98e7"} Nov 01 15:52:58.652245 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652213 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt"] Nov 01 15:52:58.652762 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652731 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerName="extract" Nov 01 15:52:58.652762 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652750 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerName="extract" Nov 01 15:52:58.652762 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652763 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerName="util" Nov 01 15:52:58.652989 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652771 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerName="util" Nov 01 15:52:58.652989 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652785 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerName="pull" Nov 01 15:52:58.652989 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652794 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerName="pull" Nov 01 15:52:58.652989 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.652907 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f7881c-54c2-4864-bb13-a8aafc6d0478" containerName="extract" Nov 01 15:52:58.676167 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.676139 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:58.694002 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.693955 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.694818 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.694029 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt"] Nov 01 15:52:58.721472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.721449 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:58.773312 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.773235 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:58.781206 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781185 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-util\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-util\") pod \"797b36d4-cdf5-4e17-a206-871d07ad0915\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " Nov 01 15:52:58.781293 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781243 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-util\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-util\") pod \"1962f9ad-1dfc-4e1e-a470-694a608a9082\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " Nov 01 15:52:58.781293 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781267 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797b36d4-cdf5-4e17-a206-871d07ad0915-kube-api-access-v8jkv\" (UniqueName: \"kubernetes.io/projected/797b36d4-cdf5-4e17-a206-871d07ad0915-kube-api-access-v8jkv\") pod \"797b36d4-cdf5-4e17-a206-871d07ad0915\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " Nov 01 15:52:58.781377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781292 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1962f9ad-1dfc-4e1e-a470-694a608a9082-kube-api-access-k7qzs\" (UniqueName: \"kubernetes.io/projected/1962f9ad-1dfc-4e1e-a470-694a608a9082-kube-api-access-k7qzs\") pod \"1962f9ad-1dfc-4e1e-a470-694a608a9082\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " Nov 01 15:52:58.781377 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781312 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-bundle\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-bundle\") pod \"797b36d4-cdf5-4e17-a206-871d07ad0915\" (UID: \"797b36d4-cdf5-4e17-a206-871d07ad0915\") " Nov 01 15:52:58.781476 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781414 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-bundle\") pod \"1962f9ad-1dfc-4e1e-a470-694a608a9082\" (UID: \"1962f9ad-1dfc-4e1e-a470-694a608a9082\") " Nov 01 15:52:58.781583 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781565 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.781645 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781597 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.781645 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781632 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2edaa19c-f839-4d7d-b845-9ae307ea4edf-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.781740 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781709 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2edaa19c-f839-4d7d-b845-9ae307ea4edf-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.781740 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781733 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.781840 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.781801 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwrm\" (UniqueName: \"kubernetes.io/projected/2edaa19c-f839-4d7d-b845-9ae307ea4edf-kube-api-access-7jwrm\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.782033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.782002 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-bundle" pod "1962f9ad-1dfc-4e1e-a470-694a608a9082" (UID: "1962f9ad-1dfc-4e1e-a470-694a608a9082"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.782119 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.782063 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-bundle" pod "797b36d4-cdf5-4e17-a206-871d07ad0915" (UID: "797b36d4-cdf5-4e17-a206-871d07ad0915"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.783846 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.783816 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1962f9ad-1dfc-4e1e-a470-694a608a9082-kube-api-access-k7qzs" pod "1962f9ad-1dfc-4e1e-a470-694a608a9082" (UID: "1962f9ad-1dfc-4e1e-a470-694a608a9082"). InnerVolumeSpecName "kube-api-access-k7qzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:58.783961 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.783850 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797b36d4-cdf5-4e17-a206-871d07ad0915-kube-api-access-v8jkv" pod "797b36d4-cdf5-4e17-a206-871d07ad0915" (UID: "797b36d4-cdf5-4e17-a206-871d07ad0915"). InnerVolumeSpecName "kube-api-access-v8jkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:58.785928 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.785910 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:58.786201 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.786180 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-util" pod "1962f9ad-1dfc-4e1e-a470-694a608a9082" (UID: "1962f9ad-1dfc-4e1e-a470-694a608a9082"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.787669 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.787647 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-util" pod "797b36d4-cdf5-4e17-a206-871d07ad0915" (UID: "797b36d4-cdf5-4e17-a206-871d07ad0915"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.801651 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.801623 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 15:52:58.801651 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.801635 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 15:52:58.801835 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.801672 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 15:52:58.801835 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.801736 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 15:52:58.801835 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.801738 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4c5x6\"" Nov 01 15:52:58.801835 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.801743 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 15:52:58.817502 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.817477 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:58.883203 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883168 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-bundle\") pod \"4e15b807-cb0c-49b1-a984-f121464d9273\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " Nov 01 15:52:58.883203 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883218 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-bundle\") pod \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " Nov 01 15:52:58.883469 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883249 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-kube-api-access-8bzwd\" (UniqueName: \"kubernetes.io/projected/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-kube-api-access-8bzwd\") pod \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " Nov 01 15:52:58.883469 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883291 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e15b807-cb0c-49b1-a984-f121464d9273-kube-api-access-m9cc6\" (UniqueName: \"kubernetes.io/projected/4e15b807-cb0c-49b1-a984-f121464d9273-kube-api-access-m9cc6\") pod \"4e15b807-cb0c-49b1-a984-f121464d9273\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " Nov 01 15:52:58.883469 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883325 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-util\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-util\") pod \"4e15b807-cb0c-49b1-a984-f121464d9273\" (UID: \"4e15b807-cb0c-49b1-a984-f121464d9273\") " Nov 01 15:52:58.883469 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883389 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-util\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-util\") pod \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\" (UID: \"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6\") " Nov 01 15:52:58.883469 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883435 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-bundle\") pod \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " Nov 01 15:52:58.883469 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883459 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-util\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-util\") pod \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " Nov 01 15:52:58.883729 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883523 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-kube-api-access-gw2w7\" (UniqueName: \"kubernetes.io/projected/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-kube-api-access-gw2w7\") pod \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\" (UID: \"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4\") " Nov 01 15:52:58.883729 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883640 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.883729 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883672 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.883729 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883708 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2edaa19c-f839-4d7d-b845-9ae307ea4edf-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.883913 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883775 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2edaa19c-f839-4d7d-b845-9ae307ea4edf-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.883913 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883799 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.883913 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883842 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwrm\" (UniqueName: \"kubernetes.io/projected/2edaa19c-f839-4d7d-b845-9ae307ea4edf-kube-api-access-7jwrm\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.883913 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883911 4911 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-util\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.884141 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883928 4911 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-util\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.884141 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883942 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jkv\" (UniqueName: \"kubernetes.io/projected/797b36d4-cdf5-4e17-a206-871d07ad0915-kube-api-access-v8jkv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.884141 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883954 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qzs\" (UniqueName: \"kubernetes.io/projected/1962f9ad-1dfc-4e1e-a470-694a608a9082-kube-api-access-k7qzs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.884141 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883963 4911 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/797b36d4-cdf5-4e17-a206-871d07ad0915-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.884141 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.883977 4911 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1962f9ad-1dfc-4e1e-a470-694a608a9082-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.884632 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.884561 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-bundle" pod "cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" (UID: "cf2a06b4-2d77-4310-a4af-bb31e1ae85d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.884632 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.884593 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-bundle" pod "4e15b807-cb0c-49b1-a984-f121464d9273" (UID: "4e15b807-cb0c-49b1-a984-f121464d9273"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.885919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.885893 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-bundle" pod "aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" (UID: "aaeeb051-99b4-4ae0-b0c1-37bcd23264d6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.886882 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.886807 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2edaa19c-f839-4d7d-b845-9ae307ea4edf-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.887156 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.887114 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e15b807-cb0c-49b1-a984-f121464d9273-kube-api-access-m9cc6" pod "4e15b807-cb0c-49b1-a984-f121464d9273" (UID: "4e15b807-cb0c-49b1-a984-f121464d9273"). InnerVolumeSpecName "kube-api-access-m9cc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:58.887156 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.887126 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-kube-api-access-gw2w7" pod "cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" (UID: "cf2a06b4-2d77-4310-a4af-bb31e1ae85d4"). InnerVolumeSpecName "kube-api-access-gw2w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:58.887706 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.887625 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.889200 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.889174 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-kube-api-access-8bzwd" pod "aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" (UID: "aaeeb051-99b4-4ae0-b0c1-37bcd23264d6"). InnerVolumeSpecName "kube-api-access-8bzwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:58.889305 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.889284 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2edaa19c-f839-4d7d-b845-9ae307ea4edf-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.890342 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.890324 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.890692 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.890670 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2edaa19c-f839-4d7d-b845-9ae307ea4edf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.891839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.891816 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-util" pod "aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" (UID: "aaeeb051-99b4-4ae0-b0c1-37bcd23264d6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.892207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.892180 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-util" pod "4e15b807-cb0c-49b1-a984-f121464d9273" (UID: "4e15b807-cb0c-49b1-a984-f121464d9273"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.896749 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.896727 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-util" pod "cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" (UID: "cf2a06b4-2d77-4310-a4af-bb31e1ae85d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:58.898071 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.898033 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwrm\" (UniqueName: \"kubernetes.io/projected/2edaa19c-f839-4d7d-b845-9ae307ea4edf-kube-api-access-7jwrm\") pod \"opendatahub-operator-controller-manager-6d5d696d67-nmwnt\" (UID: \"2edaa19c-f839-4d7d-b845-9ae307ea4edf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:58.984562 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984528 4911 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984562 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984556 4911 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984562 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984565 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bzwd\" (UniqueName: \"kubernetes.io/projected/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-kube-api-access-8bzwd\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984784 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984577 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9cc6\" (UniqueName: \"kubernetes.io/projected/4e15b807-cb0c-49b1-a984-f121464d9273-kube-api-access-m9cc6\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984784 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984587 4911 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e15b807-cb0c-49b1-a984-f121464d9273-util\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984784 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984597 4911 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaeeb051-99b4-4ae0-b0c1-37bcd23264d6-util\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984784 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984604 4911 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984784 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984612 4911 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-util\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:58.984784 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:58.984620 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2w7\" (UniqueName: \"kubernetes.io/projected/cf2a06b4-2d77-4310-a4af-bb31e1ae85d4-kube-api-access-gw2w7\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:59.059710 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.059621 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:52:59.388911 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.388882 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt"] Nov 01 15:52:59.390975 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:52:59.390949 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edaa19c_f839_4d7d_b845_9ae307ea4edf.slice/crio-5efca69cd84a73e3fbe03205edc54ce8d813e160a25f99185486f62bc53caacf WatchSource:0}: Error finding container 5efca69cd84a73e3fbe03205edc54ce8d813e160a25f99185486f62bc53caacf: Status 404 returned error can't find the container with id 5efca69cd84a73e3fbe03205edc54ce8d813e160a25f99185486f62bc53caacf Nov 01 15:52:59.492461 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.492431 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" Nov 01 15:52:59.492857 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.492433 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jwbfv" event={"ID":"4e15b807-cb0c-49b1-a984-f121464d9273","Type":"ContainerDied","Data":"1ba074c0e8912595ab448fe998e1c903b8ff8ed467ce72443b83d1f77198a6e0"} Nov 01 15:52:59.492857 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.492589 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ba074c0e8912595ab448fe998e1c903b8ff8ed467ce72443b83d1f77198a6e0" Nov 01 15:52:59.494431 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.494408 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" event={"ID":"cf2a06b4-2d77-4310-a4af-bb31e1ae85d4","Type":"ContainerDied","Data":"a163e5b09110bf8eca41e19b7fa7195710775ad80fec7f53bbee4e07106b2d09"} Nov 01 15:52:59.494566 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.494449 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a163e5b09110bf8eca41e19b7fa7195710775ad80fec7f53bbee4e07106b2d09" Nov 01 15:52:59.494566 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.494417 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdn9lg" Nov 01 15:52:59.496397 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.496380 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" Nov 01 15:52:59.496490 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.496402 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nj7lz" event={"ID":"aaeeb051-99b4-4ae0-b0c1-37bcd23264d6","Type":"ContainerDied","Data":"64100bdf88aa80c1b97012a316d57e94361bf146b60de9bd45ecafa9ce3e7e4c"} Nov 01 15:52:59.496490 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.496443 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64100bdf88aa80c1b97012a316d57e94361bf146b60de9bd45ecafa9ce3e7e4c" Nov 01 15:52:59.497911 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.497885 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" event={"ID":"2edaa19c-f839-4d7d-b845-9ae307ea4edf","Type":"ContainerStarted","Data":"5efca69cd84a73e3fbe03205edc54ce8d813e160a25f99185486f62bc53caacf"} Nov 01 15:52:59.499922 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.499897 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" Nov 01 15:52:59.500017 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.499900 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnng89" event={"ID":"1962f9ad-1dfc-4e1e-a470-694a608a9082","Type":"ContainerDied","Data":"498c763e54df8d6a321d26aeb4072df357b2c884fe4f62a47fcd3cee1d44144c"} Nov 01 15:52:59.500017 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.500008 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="498c763e54df8d6a321d26aeb4072df357b2c884fe4f62a47fcd3cee1d44144c" Nov 01 15:52:59.502076 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.502030 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" event={"ID":"797b36d4-cdf5-4e17-a206-871d07ad0915","Type":"ContainerDied","Data":"90bd1078b777cc754dfae00c392cd2f7126fef1be307c9615de39f7b4a1ca2f3"} Nov 01 15:52:59.502166 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.502094 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awkqjx" Nov 01 15:52:59.502166 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.502095 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90bd1078b777cc754dfae00c392cd2f7126fef1be307c9615de39f7b4a1ca2f3" Nov 01 15:52:59.687687 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.687655 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:52:59.790149 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.790114 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0df6725-a199-496f-8160-0867c9b169e0-kube-api-access-gclpn\" (UniqueName: \"kubernetes.io/projected/e0df6725-a199-496f-8160-0867c9b169e0-kube-api-access-gclpn\") pod \"e0df6725-a199-496f-8160-0867c9b169e0\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " Nov 01 15:52:59.790331 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.790197 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-bundle\") pod \"e0df6725-a199-496f-8160-0867c9b169e0\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " Nov 01 15:52:59.790331 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.790220 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-util\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-util\") pod \"e0df6725-a199-496f-8160-0867c9b169e0\" (UID: \"e0df6725-a199-496f-8160-0867c9b169e0\") " Nov 01 15:52:59.790596 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.790571 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-bundle" pod "e0df6725-a199-496f-8160-0867c9b169e0" (UID: "e0df6725-a199-496f-8160-0867c9b169e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:59.792348 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.792324 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0df6725-a199-496f-8160-0867c9b169e0-kube-api-access-gclpn" pod "e0df6725-a199-496f-8160-0867c9b169e0" (UID: "e0df6725-a199-496f-8160-0867c9b169e0"). InnerVolumeSpecName "kube-api-access-gclpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:59.795728 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.795697 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-util" pod "e0df6725-a199-496f-8160-0867c9b169e0" (UID: "e0df6725-a199-496f-8160-0867c9b169e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:59.894290 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.894192 4911 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-bundle\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:59.894290 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.894223 4911 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0df6725-a199-496f-8160-0867c9b169e0-util\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:59.894290 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:52:59.894237 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gclpn\" (UniqueName: \"kubernetes.io/projected/e0df6725-a199-496f-8160-0867c9b169e0-kube-api-access-gclpn\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:00.511292 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:00.511253 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" event={"ID":"e0df6725-a199-496f-8160-0867c9b169e0","Type":"ContainerDied","Data":"fc55815f44b93d1e309a37aed6c136b81abc34ab6546c10f3bbf8ef867f40c5d"} Nov 01 15:53:00.511292 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:00.511289 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qnzm9" Nov 01 15:53:00.511805 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:00.511303 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc55815f44b93d1e309a37aed6c136b81abc34ab6546c10f3bbf8ef867f40c5d" Nov 01 15:53:02.523227 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:02.523191 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" event={"ID":"2edaa19c-f839-4d7d-b845-9ae307ea4edf","Type":"ContainerStarted","Data":"4e1499df008ce184e48f550417858152d6c7cc6429776424b585859380cf5aff"} Nov 01 15:53:03.528971 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:03.528933 4911 generic.go:356] "Generic (PLEG): container finished" podID="2edaa19c-f839-4d7d-b845-9ae307ea4edf" containerID="4e1499df008ce184e48f550417858152d6c7cc6429776424b585859380cf5aff" exitCode=0 Nov 01 15:53:03.529432 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:03.529014 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" event={"ID":"2edaa19c-f839-4d7d-b845-9ae307ea4edf","Type":"ContainerDied","Data":"4e1499df008ce184e48f550417858152d6c7cc6429776424b585859380cf5aff"} Nov 01 15:53:04.535595 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:04.535557 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" event={"ID":"2edaa19c-f839-4d7d-b845-9ae307ea4edf","Type":"ContainerStarted","Data":"8116b0fc53a24dac41c212dafaabc2d0f03357767faf263984bab23c231ba4f8"} Nov 01 15:53:04.535961 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:04.535730 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:53:04.571130 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:04.571075 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" podStartSLOduration=2.762381201 podStartE2EDuration="6.571062183s" totalImagesPullingTime="3.398184717s" totalInitContainerRuntime="410.496265ms" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:04.568505893 +0000 UTC m=+324.712960534" watchObservedRunningTime="2026-11-01 15:53:04.571062183 +0000 UTC m=+324.715516824" Nov 01 15:53:10.864815 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:53:10.864774 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:53:15.542965 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:15.542933 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-nmwnt" Nov 01 15:53:17.257911 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.257874 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz"] Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258221 4911 cpu_manager.go:414] "containerMap: removing container" podUID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerName="pull" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258233 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerName="pull" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258255 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4e15b807-cb0c-49b1-a984-f121464d9273" containerName="pull" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258260 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15b807-cb0c-49b1-a984-f121464d9273" containerName="pull" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258270 4911 cpu_manager.go:414] "containerMap: removing container" podUID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerName="pull" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258275 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerName="pull" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258285 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e0df6725-a199-496f-8160-0867c9b169e0" containerName="extract" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258290 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0df6725-a199-496f-8160-0867c9b169e0" containerName="extract" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258296 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerName="util" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258301 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerName="util" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258311 4911 cpu_manager.go:414] "containerMap: removing container" podUID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerName="extract" Nov 01 15:53:17.258315 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258315 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258326 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e0df6725-a199-496f-8160-0867c9b169e0" containerName="pull" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258330 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0df6725-a199-496f-8160-0867c9b169e0" containerName="pull" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258336 4911 cpu_manager.go:414] "containerMap: removing container" podUID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258341 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258348 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerName="pull" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258352 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerName="pull" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258359 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e0df6725-a199-496f-8160-0867c9b169e0" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258363 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0df6725-a199-496f-8160-0867c9b169e0" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258371 4911 cpu_manager.go:414] "containerMap: removing container" podUID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258375 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258380 4911 cpu_manager.go:414] "containerMap: removing container" podUID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258384 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258389 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4e15b807-cb0c-49b1-a984-f121464d9273" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258393 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15b807-cb0c-49b1-a984-f121464d9273" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258400 4911 cpu_manager.go:414] "containerMap: removing container" podUID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258404 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerName="util" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258412 4911 cpu_manager.go:414] "containerMap: removing container" podUID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerName="pull" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258416 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerName="pull" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258422 4911 cpu_manager.go:414] "containerMap: removing container" podUID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258428 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258434 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258439 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258444 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4e15b807-cb0c-49b1-a984-f121464d9273" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258453 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e15b807-cb0c-49b1-a984-f121464d9273" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258504 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0df6725-a199-496f-8160-0867c9b169e0" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258512 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="797b36d4-cdf5-4e17-a206-871d07ad0915" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258519 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2a06b4-2d77-4310-a4af-bb31e1ae85d4" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258525 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1962f9ad-1dfc-4e1e-a470-694a608a9082" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258531 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaeeb051-99b4-4ae0-b0c1-37bcd23264d6" containerName="extract" Nov 01 15:53:17.258650 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.258536 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e15b807-cb0c-49b1-a984-f121464d9273" containerName="extract" Nov 01 15:53:17.265963 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.265937 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" Nov 01 15:53:17.271492 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.271466 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-g2rw7\"" Nov 01 15:53:17.271647 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.271616 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 15:53:17.272107 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.272080 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz"] Nov 01 15:53:17.272388 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.272372 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 15:53:17.345161 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.345126 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv655\" (UniqueName: \"kubernetes.io/projected/c11ba1b9-a4fa-45ea-adbc-b13aea773ca0-kube-api-access-cv655\") pod \"authorino-operator-7b7bb4bc77-8m4nz\" (UID: \"c11ba1b9-a4fa-45ea-adbc-b13aea773ca0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" Nov 01 15:53:17.446400 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.446357 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv655\" (UniqueName: \"kubernetes.io/projected/c11ba1b9-a4fa-45ea-adbc-b13aea773ca0-kube-api-access-cv655\") pod \"authorino-operator-7b7bb4bc77-8m4nz\" (UID: \"c11ba1b9-a4fa-45ea-adbc-b13aea773ca0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" Nov 01 15:53:17.455207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.455175 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv655\" (UniqueName: \"kubernetes.io/projected/c11ba1b9-a4fa-45ea-adbc-b13aea773ca0-kube-api-access-cv655\") pod \"authorino-operator-7b7bb4bc77-8m4nz\" (UID: \"c11ba1b9-a4fa-45ea-adbc-b13aea773ca0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" Nov 01 15:53:17.579090 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.578988 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" Nov 01 15:53:17.852931 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:17.852901 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz"] Nov 01 15:53:17.855444 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:53:17.855410 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11ba1b9_a4fa_45ea_adbc_b13aea773ca0.slice/crio-037334e3efe18bc544a3623cdf9679e4dfb5a5d1060d26681e29bb777d037c15 WatchSource:0}: Error finding container 037334e3efe18bc544a3623cdf9679e4dfb5a5d1060d26681e29bb777d037c15: Status 404 returned error can't find the container with id 037334e3efe18bc544a3623cdf9679e4dfb5a5d1060d26681e29bb777d037c15 Nov 01 15:53:18.602032 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:18.601997 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" event={"ID":"c11ba1b9-a4fa-45ea-adbc-b13aea773ca0","Type":"ContainerStarted","Data":"037334e3efe18bc544a3623cdf9679e4dfb5a5d1060d26681e29bb777d037c15"} Nov 01 15:53:20.046670 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.046636 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw"] Nov 01 15:53:20.050206 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.050181 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" Nov 01 15:53:20.056467 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.056445 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 15:53:20.056602 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.056498 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-npffb\"" Nov 01 15:53:20.060475 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.060447 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw"] Nov 01 15:53:20.168521 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.168483 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/59d8fcb3-1620-414d-a8e4-104c9d63e26d-kube-api-access-q7hq6\") pod \"dns-operator-controller-manager-8557cbdb84-v26kw\" (UID: \"59d8fcb3-1620-414d-a8e4-104c9d63e26d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" Nov 01 15:53:20.269720 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.269675 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/59d8fcb3-1620-414d-a8e4-104c9d63e26d-kube-api-access-q7hq6\") pod \"dns-operator-controller-manager-8557cbdb84-v26kw\" (UID: \"59d8fcb3-1620-414d-a8e4-104c9d63e26d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" Nov 01 15:53:20.284231 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.284185 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/59d8fcb3-1620-414d-a8e4-104c9d63e26d-kube-api-access-q7hq6\") pod \"dns-operator-controller-manager-8557cbdb84-v26kw\" (UID: \"59d8fcb3-1620-414d-a8e4-104c9d63e26d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" Nov 01 15:53:20.366110 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.366008 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" Nov 01 15:53:20.650294 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:20.650190 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw"] Nov 01 15:53:20.653188 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:53:20.653155 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d8fcb3_1620_414d_a8e4_104c9d63e26d.slice/crio-97f492c9acb6dcd2f54531624c30d1c41ee6ae30a030211ce53c536cb0dfbced WatchSource:0}: Error finding container 97f492c9acb6dcd2f54531624c30d1c41ee6ae30a030211ce53c536cb0dfbced: Status 404 returned error can't find the container with id 97f492c9acb6dcd2f54531624c30d1c41ee6ae30a030211ce53c536cb0dfbced Nov 01 15:53:21.619678 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:21.619646 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" event={"ID":"59d8fcb3-1620-414d-a8e4-104c9d63e26d","Type":"ContainerStarted","Data":"97f492c9acb6dcd2f54531624c30d1c41ee6ae30a030211ce53c536cb0dfbced"} Nov 01 15:53:21.621215 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:21.621186 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" event={"ID":"c11ba1b9-a4fa-45ea-adbc-b13aea773ca0","Type":"ContainerStarted","Data":"2cca8210b3fead8fae092f6c9cc49f7780c055cd1a7d162e2d5825608377774f"} Nov 01 15:53:21.621340 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:21.621326 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" Nov 01 15:53:21.640686 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:21.640643 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" podStartSLOduration=1.36356056 podStartE2EDuration="4.640629749s" totalImagesPullingTime="3.277069189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:21.639216808 +0000 UTC m=+341.783671450" watchObservedRunningTime="2026-11-01 15:53:21.640629749 +0000 UTC m=+341.785084389" Nov 01 15:53:23.259895 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.259859 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq"] Nov 01 15:53:23.263274 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.263256 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" Nov 01 15:53:23.277897 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.277875 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bg7fj\"" Nov 01 15:53:23.307027 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.307002 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq"] Nov 01 15:53:23.395555 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.395520 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgcc\" (UniqueName: \"kubernetes.io/projected/3f3b6fab-75bb-416f-a042-125cf512aa4e-kube-api-access-krgcc\") pod \"limitador-operator-controller-manager-5778bb4449-4ntsq\" (UID: \"3f3b6fab-75bb-416f-a042-125cf512aa4e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" Nov 01 15:53:23.496219 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.496147 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgcc\" (UniqueName: \"kubernetes.io/projected/3f3b6fab-75bb-416f-a042-125cf512aa4e-kube-api-access-krgcc\") pod \"limitador-operator-controller-manager-5778bb4449-4ntsq\" (UID: \"3f3b6fab-75bb-416f-a042-125cf512aa4e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" Nov 01 15:53:23.506916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.506887 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgcc\" (UniqueName: \"kubernetes.io/projected/3f3b6fab-75bb-416f-a042-125cf512aa4e-kube-api-access-krgcc\") pod \"limitador-operator-controller-manager-5778bb4449-4ntsq\" (UID: \"3f3b6fab-75bb-416f-a042-125cf512aa4e\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" Nov 01 15:53:23.575598 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.575525 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" Nov 01 15:53:23.634915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.634879 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" event={"ID":"59d8fcb3-1620-414d-a8e4-104c9d63e26d","Type":"ContainerStarted","Data":"8a507073c9efaa0237927f96875292c9eadf5552840150c57df05f04cc1b4a41"} Nov 01 15:53:23.635145 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.635121 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" Nov 01 15:53:23.655581 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.655529 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" podStartSLOduration=1.6854333700000002 podStartE2EDuration="3.655515156s" totalImagesPullingTime="1.970081786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:23.654764994 +0000 UTC m=+343.799219634" watchObservedRunningTime="2026-11-01 15:53:23.655515156 +0000 UTC m=+343.799969798" Nov 01 15:53:23.855499 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:23.855469 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq"] Nov 01 15:53:23.857035 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:53:23.857003 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3b6fab_75bb_416f_a042_125cf512aa4e.slice/crio-ebc7f87c697ef9aa5731fa087c2c0730965ef56e253b5d11c06039ccaa28fd0e WatchSource:0}: Error finding container ebc7f87c697ef9aa5731fa087c2c0730965ef56e253b5d11c06039ccaa28fd0e: Status 404 returned error can't find the container with id ebc7f87c697ef9aa5731fa087c2c0730965ef56e253b5d11c06039ccaa28fd0e Nov 01 15:53:24.640686 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:24.640649 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" event={"ID":"3f3b6fab-75bb-416f-a042-125cf512aa4e","Type":"ContainerStarted","Data":"ebc7f87c697ef9aa5731fa087c2c0730965ef56e253b5d11c06039ccaa28fd0e"} Nov 01 15:53:26.652726 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:26.652679 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" event={"ID":"3f3b6fab-75bb-416f-a042-125cf512aa4e","Type":"ContainerStarted","Data":"0208d8355f251052f6a138a507e01e19a4f1dff2ed66de51b71812004b82b0d6"} Nov 01 15:53:26.653236 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:26.652760 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" Nov 01 15:53:26.674359 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:26.674314 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" podStartSLOduration=1.698323249 podStartE2EDuration="3.674302504s" totalImagesPullingTime="1.975979255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:26.67422002 +0000 UTC m=+346.818674661" watchObservedRunningTime="2026-11-01 15:53:26.674302504 +0000 UTC m=+346.818757142" Nov 01 15:53:29.640884 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.640849 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws"] Nov 01 15:53:29.644360 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.644339 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:29.648570 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.648552 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r57c6\"" Nov 01 15:53:29.655524 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.655499 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws"] Nov 01 15:53:29.743553 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.743515 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28kb\" (UniqueName: \"kubernetes.io/projected/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-kube-api-access-n28kb\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xw5ws\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:29.743714 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.743596 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xw5ws\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:29.844222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.844179 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28kb\" (UniqueName: \"kubernetes.io/projected/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-kube-api-access-n28kb\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xw5ws\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:29.844415 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.844234 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xw5ws\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:29.844638 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.844613 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xw5ws\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:29.853506 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.853473 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28kb\" (UniqueName: \"kubernetes.io/projected/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-kube-api-access-n28kb\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xw5ws\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:29.957371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:29.957288 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:30.233482 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:30.233458 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws"] Nov 01 15:53:30.235512 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:53:30.235474 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a7bb1d_7e53_40e9_9fb6_ab197013a6c6.slice/crio-b546578bcdfb6c6767ae04287d36191b324a7d78ab0766d81bdf5d46df1e41b1 WatchSource:0}: Error finding container b546578bcdfb6c6767ae04287d36191b324a7d78ab0766d81bdf5d46df1e41b1: Status 404 returned error can't find the container with id b546578bcdfb6c6767ae04287d36191b324a7d78ab0766d81bdf5d46df1e41b1 Nov 01 15:53:30.679756 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:30.679723 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" event={"ID":"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6","Type":"ContainerStarted","Data":"b546578bcdfb6c6767ae04287d36191b324a7d78ab0766d81bdf5d46df1e41b1"} Nov 01 15:53:32.628005 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:32.627971 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8m4nz" Nov 01 15:53:34.642698 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:34.642665 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-v26kw" Nov 01 15:53:34.705782 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:34.705741 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" event={"ID":"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6","Type":"ContainerStarted","Data":"f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e"} Nov 01 15:53:34.705971 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:34.705927 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:34.725904 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:34.725855 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" podStartSLOduration=1.935804324 podStartE2EDuration="5.725841723s" totalImagesPullingTime="3.790037399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:34.723813566 +0000 UTC m=+354.868268205" watchObservedRunningTime="2026-11-01 15:53:34.725841723 +0000 UTC m=+354.870296366" Nov 01 15:53:37.664582 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.664550 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4ntsq" Nov 01 15:53:37.863239 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.863198 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k"] Nov 01 15:53:37.872684 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.872656 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.885510 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.885486 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k"] Nov 01 15:53:37.904527 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904492 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904558 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904630 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4tt\" (UniqueName: \"kubernetes.io/projected/1fa9f79a-f263-46fd-8556-5fc1027e3b61-kube-api-access-cj4tt\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904781 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904699 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904781 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904760 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904877 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904815 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904936 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904894 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904991 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904934 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.904991 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.904972 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:37.920258 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:37.920205 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bt8fx\"" Nov 01 15:53:38.005730 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005701 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.005916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005738 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.005916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005765 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.005916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005785 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.005916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005814 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.005916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005857 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.005916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005877 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4tt\" (UniqueName: \"kubernetes.io/projected/1fa9f79a-f263-46fd-8556-5fc1027e3b61-kube-api-access-cj4tt\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.005916 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005902 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.006317 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.005930 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.006317 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.006217 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.006418 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.006348 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.006418 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.006366 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.006418 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.006399 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.006837 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.006813 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.008358 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.008338 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.008477 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.008461 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.049875 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.049849 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa9f79a-f263-46fd-8556-5fc1027e3b61-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.050992 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.050970 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4tt\" (UniqueName: \"kubernetes.io/projected/1fa9f79a-f263-46fd-8556-5fc1027e3b61-kube-api-access-cj4tt\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k\" (UID: \"1fa9f79a-f263-46fd-8556-5fc1027e3b61\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.186271 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.186189 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:38.474034 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.473988 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k"] Nov 01 15:53:38.476291 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:53:38.476262 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa9f79a_f263_46fd_8556_5fc1027e3b61.slice/crio-cfa97f1fc424f1345e844555ca9677050fe253e7088ea85cde64e55a3c0e32d7 WatchSource:0}: Error finding container cfa97f1fc424f1345e844555ca9677050fe253e7088ea85cde64e55a3c0e32d7: Status 404 returned error can't find the container with id cfa97f1fc424f1345e844555ca9677050fe253e7088ea85cde64e55a3c0e32d7 Nov 01 15:53:38.478475 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.478442 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:53:38.478529 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.478510 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:53:38.478568 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.478544 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:53:38.729732 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.729617 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" event={"ID":"1fa9f79a-f263-46fd-8556-5fc1027e3b61","Type":"ContainerStarted","Data":"aceb65ce662418c42f91d758d8b08c2f004e60b6bd137c848ee00b306b34e085"} Nov 01 15:53:38.729732 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.729681 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" event={"ID":"1fa9f79a-f263-46fd-8556-5fc1027e3b61","Type":"ContainerStarted","Data":"cfa97f1fc424f1345e844555ca9677050fe253e7088ea85cde64e55a3c0e32d7"} Nov 01 15:53:38.763705 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:38.763656 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" podStartSLOduration=1.7636408719999999 podStartE2EDuration="1.763640872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:38.76252202 +0000 UTC m=+358.906976662" watchObservedRunningTime="2026-11-01 15:53:38.763640872 +0000 UTC m=+358.908095512" Nov 01 15:53:39.187224 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.187187 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:39.192105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.192078 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:39.321310 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.321273 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz"] Nov 01 15:53:39.325207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.325190 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.335770 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.335740 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz"] Nov 01 15:53:39.363336 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.363303 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 15:53:39.363488 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.363365 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 15:53:39.408476 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.408443 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mmpp7\"" Nov 01 15:53:39.419004 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.418962 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7009d3f-9ea5-49e9-a75b-45064c82e923-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-xchfz\" (UID: \"c7009d3f-9ea5-49e9-a75b-45064c82e923\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.419179 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.419051 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxvb\" (UniqueName: \"kubernetes.io/projected/c7009d3f-9ea5-49e9-a75b-45064c82e923-kube-api-access-bmxvb\") pod \"cert-manager-cainjector-5c57bbd4fc-xchfz\" (UID: \"c7009d3f-9ea5-49e9-a75b-45064c82e923\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.519805 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.519772 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7009d3f-9ea5-49e9-a75b-45064c82e923-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-xchfz\" (UID: \"c7009d3f-9ea5-49e9-a75b-45064c82e923\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.519990 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.519830 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxvb\" (UniqueName: \"kubernetes.io/projected/c7009d3f-9ea5-49e9-a75b-45064c82e923-kube-api-access-bmxvb\") pod \"cert-manager-cainjector-5c57bbd4fc-xchfz\" (UID: \"c7009d3f-9ea5-49e9-a75b-45064c82e923\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.534986 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.534957 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7009d3f-9ea5-49e9-a75b-45064c82e923-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-xchfz\" (UID: \"c7009d3f-9ea5-49e9-a75b-45064c82e923\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.535220 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.535194 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxvb\" (UniqueName: \"kubernetes.io/projected/c7009d3f-9ea5-49e9-a75b-45064c82e923-kube-api-access-bmxvb\") pod \"cert-manager-cainjector-5c57bbd4fc-xchfz\" (UID: \"c7009d3f-9ea5-49e9-a75b-45064c82e923\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.539604 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.539579 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rzkj9"] Nov 01 15:53:39.543915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.543894 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" Nov 01 15:53:39.551608 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.551586 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rzkj9"] Nov 01 15:53:39.620354 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.620324 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lxq\" (UniqueName: \"kubernetes.io/projected/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92-kube-api-access-48lxq\") pod \"authorino-57f7f8659f-rzkj9\" (UID: \"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92\") " pod="kuadrant-system/authorino-57f7f8659f-rzkj9" Nov 01 15:53:39.620863 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.620833 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bjtb4\"" Nov 01 15:53:39.648964 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.648933 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" Nov 01 15:53:39.721734 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.721702 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lxq\" (UniqueName: \"kubernetes.io/projected/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92-kube-api-access-48lxq\") pod \"authorino-57f7f8659f-rzkj9\" (UID: \"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92\") " pod="kuadrant-system/authorino-57f7f8659f-rzkj9" Nov 01 15:53:39.733752 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.733722 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lxq\" (UniqueName: \"kubernetes.io/projected/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92-kube-api-access-48lxq\") pod \"authorino-57f7f8659f-rzkj9\" (UID: \"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92\") " pod="kuadrant-system/authorino-57f7f8659f-rzkj9" Nov 01 15:53:39.735831 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.735810 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:39.736854 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.736837 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k" Nov 01 15:53:39.862197 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.862121 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" Nov 01 15:53:39.929085 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.928946 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8zcbq"] Nov 01 15:53:39.933717 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.933677 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz"] Nov 01 15:53:39.933807 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.933787 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" Nov 01 15:53:39.945257 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:39.945232 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8zcbq"] Nov 01 15:53:40.031378 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.031345 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8k4\" (UniqueName: \"kubernetes.io/projected/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5-kube-api-access-tg8k4\") pod \"authorino-56c6c9ddd5-8zcbq\" (UID: \"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5\") " pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" Nov 01 15:53:40.131952 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.131885 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8k4\" (UniqueName: \"kubernetes.io/projected/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5-kube-api-access-tg8k4\") pod \"authorino-56c6c9ddd5-8zcbq\" (UID: \"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5\") " pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" Nov 01 15:53:40.139832 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.139809 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rzkj9"] Nov 01 15:53:40.141600 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:53:40.141567 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod294f7f0c_d8f3_4001_8ea4_fdeef3a7ab92.slice/crio-1e6ec5bfa973b2bb562c41e01ae1c1cf7bc313ea3a916d33f8757ec6c66d3b35 WatchSource:0}: Error finding container 1e6ec5bfa973b2bb562c41e01ae1c1cf7bc313ea3a916d33f8757ec6c66d3b35: Status 404 returned error can't find the container with id 1e6ec5bfa973b2bb562c41e01ae1c1cf7bc313ea3a916d33f8757ec6c66d3b35 Nov 01 15:53:40.145842 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.145824 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8k4\" (UniqueName: \"kubernetes.io/projected/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5-kube-api-access-tg8k4\") pod \"authorino-56c6c9ddd5-8zcbq\" (UID: \"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5\") " pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" Nov 01 15:53:40.250750 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.250717 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" Nov 01 15:53:40.522617 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.522594 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8zcbq"] Nov 01 15:53:40.524448 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:53:40.524417 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb74b4de9_2b48_4cda_b92b_11ceaa4bbda5.slice/crio-7af0f7deddc7d9020a2d41e2974cb1aa548c19023d1cf720e63f2170bcd68455 WatchSource:0}: Error finding container 7af0f7deddc7d9020a2d41e2974cb1aa548c19023d1cf720e63f2170bcd68455: Status 404 returned error can't find the container with id 7af0f7deddc7d9020a2d41e2974cb1aa548c19023d1cf720e63f2170bcd68455 Nov 01 15:53:40.741699 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.741660 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" event={"ID":"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5","Type":"ContainerStarted","Data":"7af0f7deddc7d9020a2d41e2974cb1aa548c19023d1cf720e63f2170bcd68455"} Nov 01 15:53:40.742900 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.742868 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" event={"ID":"c7009d3f-9ea5-49e9-a75b-45064c82e923","Type":"ContainerStarted","Data":"4e81d408db1543148d21ebd54d21b01261f59d202705183b8768c9847cd4499e"} Nov 01 15:53:40.744074 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:40.744031 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" event={"ID":"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92","Type":"ContainerStarted","Data":"1e6ec5bfa973b2bb562c41e01ae1c1cf7bc313ea3a916d33f8757ec6c66d3b35"} Nov 01 15:53:40.865071 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:53:40.864975 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:53:42.762089 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:42.761420 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" event={"ID":"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92","Type":"ContainerStarted","Data":"e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca"} Nov 01 15:53:42.763818 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:42.763784 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" event={"ID":"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5","Type":"ContainerStarted","Data":"ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd"} Nov 01 15:53:42.779705 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:42.779654 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" podStartSLOduration=1.329902789 podStartE2EDuration="3.779637966s" totalImagesPullingTime="2.449735177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:42.77786246 +0000 UTC m=+362.922317118" watchObservedRunningTime="2026-11-01 15:53:42.779637966 +0000 UTC m=+362.924092609" Nov 01 15:53:42.801269 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:42.801207 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" podStartSLOduration=1.683882982 podStartE2EDuration="3.80118843s" totalImagesPullingTime="2.117305448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:42.80039172 +0000 UTC m=+362.944846398" watchObservedRunningTime="2026-11-01 15:53:42.80118843 +0000 UTC m=+362.945643072" Nov 01 15:53:42.823863 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:42.823823 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rzkj9"] Nov 01 15:53:43.774251 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:43.774204 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" event={"ID":"c7009d3f-9ea5-49e9-a75b-45064c82e923","Type":"ContainerStarted","Data":"ec1be79d320cf32f38b653751f8231ea2047a790627b5830595594fc3683f873"} Nov 01 15:53:43.790474 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:43.790422 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-xchfz" podStartSLOduration=1.500758282 podStartE2EDuration="4.790405915s" totalImagesPullingTime="3.289647633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:43.78998586 +0000 UTC m=+363.934440503" watchObservedRunningTime="2026-11-01 15:53:43.790405915 +0000 UTC m=+363.934860554" Nov 01 15:53:44.778612 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:44.778571 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" podUID="294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92" containerName="authorino" containerID="cri-o://e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca" gracePeriod=30 Nov 01 15:53:45.028557 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.028534 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" Nov 01 15:53:45.081296 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.081219 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92-kube-api-access-48lxq\" (UniqueName: \"kubernetes.io/projected/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92-kube-api-access-48lxq\") pod \"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92\" (UID: \"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92\") " Nov 01 15:53:45.083379 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.083351 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92-kube-api-access-48lxq" pod "294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92" (UID: "294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92"). InnerVolumeSpecName "kube-api-access-48lxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:53:45.182830 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.182792 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48lxq\" (UniqueName: \"kubernetes.io/projected/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92-kube-api-access-48lxq\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:45.712906 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.712868 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:53:45.785698 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.785656 4911 generic.go:356] "Generic (PLEG): container finished" podID="294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92" containerID="e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca" exitCode=0 Nov 01 15:53:45.785698 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.785695 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" Nov 01 15:53:45.786238 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.785724 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" event={"ID":"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92","Type":"ContainerDied","Data":"e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca"} Nov 01 15:53:45.786238 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.785768 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rzkj9" event={"ID":"294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92","Type":"ContainerDied","Data":"1e6ec5bfa973b2bb562c41e01ae1c1cf7bc313ea3a916d33f8757ec6c66d3b35"} Nov 01 15:53:45.786238 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.785788 4911 scope.go:122] "RemoveContainer" containerID="e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca" Nov 01 15:53:45.794670 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.794654 4911 scope.go:122] "RemoveContainer" containerID="e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca" Nov 01 15:53:45.794925 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:53:45.794905 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca\": container with ID starting with e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca not found: ID does not exist" containerID="e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca" Nov 01 15:53:45.795019 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.794933 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca"} err="failed to get container status \"e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca\": rpc error: code = NotFound desc = could not find container \"e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca\": container with ID starting with e7fc09c723bcc5467041b0cedbffc3e9a8909e5f0006689c29141ab40e9e61ca not found: ID does not exist" Nov 01 15:53:45.821010 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.820971 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rzkj9"] Nov 01 15:53:45.827311 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.827288 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rzkj9"] Nov 01 15:53:45.969822 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:53:45.969747 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92" path="/var/lib/kubelet/pods/294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92/volumes" Nov 01 15:54:02.250214 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.250182 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws"] Nov 01 15:54:02.250750 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.250369 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" podUID="d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" containerName="manager" containerID="cri-o://f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e" gracePeriod=10 Nov 01 15:54:02.504749 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.504683 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:54:02.637471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.637436 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-kube-api-access-n28kb\" (UniqueName: \"kubernetes.io/projected/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-kube-api-access-n28kb\") pod \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " Nov 01 15:54:02.637652 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.637511 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-extensions-socket-volume\") pod \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\" (UID: \"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6\") " Nov 01 15:54:02.637868 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.637843 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-extensions-socket-volume" pod "d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" (UID: "d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:54:02.639579 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.639552 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-kube-api-access-n28kb" pod "d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" (UID: "d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6"). InnerVolumeSpecName "kube-api-access-n28kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:54:02.738735 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.738698 4911 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-extensions-socket-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:02.738735 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.738731 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28kb\" (UniqueName: \"kubernetes.io/projected/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6-kube-api-access-n28kb\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:02.870173 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.870087 4911 generic.go:356] "Generic (PLEG): container finished" podID="d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" containerID="f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e" exitCode=0 Nov 01 15:54:02.870173 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.870139 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" Nov 01 15:54:02.870173 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.870161 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" event={"ID":"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6","Type":"ContainerDied","Data":"f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e"} Nov 01 15:54:02.870401 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.870202 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws" event={"ID":"d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6","Type":"ContainerDied","Data":"b546578bcdfb6c6767ae04287d36191b324a7d78ab0766d81bdf5d46df1e41b1"} Nov 01 15:54:02.870401 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.870225 4911 scope.go:122] "RemoveContainer" containerID="f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e" Nov 01 15:54:02.886278 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.886257 4911 scope.go:122] "RemoveContainer" containerID="f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e" Nov 01 15:54:02.886614 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:54:02.886594 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e\": container with ID starting with f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e not found: ID does not exist" containerID="f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e" Nov 01 15:54:02.886667 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.886622 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e"} err="failed to get container status \"f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e\": rpc error: code = NotFound desc = could not find container \"f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e\": container with ID starting with f34b2895b922661f5e6002025ab8ad6a3891845dd5f4837b687ffb2c3ff0ab9e not found: ID does not exist" Nov 01 15:54:02.909723 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.909688 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws"] Nov 01 15:54:02.923090 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:02.923062 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xw5ws"] Nov 01 15:54:03.964545 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:03.964507 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" path="/var/lib/kubelet/pods/d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6/volumes" Nov 01 15:54:10.864621 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:54:10.864582 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:54:19.829292 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.829249 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-njj26"] Nov 01 15:54:19.829966 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.829798 4911 cpu_manager.go:414] "containerMap: removing container" podUID="294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92" containerName="authorino" Nov 01 15:54:19.829966 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.829820 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92" containerName="authorino" Nov 01 15:54:19.829966 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.829859 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" containerName="manager" Nov 01 15:54:19.829966 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.829867 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" containerName="manager" Nov 01 15:54:19.830225 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.829969 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="294f7f0c-d8f3-4001-8ea4-fdeef3a7ab92" containerName="authorino" Nov 01 15:54:19.830225 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.829985 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a7bb1d-7e53-40e9-9fb6-ab197013a6c6" containerName="manager" Nov 01 15:54:19.832639 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.832618 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.837610 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.837586 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 15:54:19.837610 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.837612 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 15:54:19.839114 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.839091 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 15:54:19.839229 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.839123 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 15:54:19.839229 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.839149 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 15:54:19.839926 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.839905 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kqkln\"" Nov 01 15:54:19.842336 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.842314 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-njj26"] Nov 01 15:54:19.892706 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.892674 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-webhook-cert\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.892865 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.892789 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqhl\" (UniqueName: \"kubernetes.io/projected/f367cd26-cc47-4cdc-96f4-e84a155546c9-kube-api-access-tcqhl\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.892865 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.892844 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-metrics-certs\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.993484 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.993446 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqhl\" (UniqueName: \"kubernetes.io/projected/f367cd26-cc47-4cdc-96f4-e84a155546c9-kube-api-access-tcqhl\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.993657 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.993504 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-metrics-certs\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.993657 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.993533 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-webhook-cert\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.996065 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.996025 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-webhook-cert\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:19.996179 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:19.996082 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-metrics-certs\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:20.001765 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.001735 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqhl\" (UniqueName: \"kubernetes.io/projected/f367cd26-cc47-4cdc-96f4-e84a155546c9-kube-api-access-tcqhl\") pod \"maas-controller-579cf7c6d-njj26\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:20.145539 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.145444 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:20.149320 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.149295 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-646677ff99-h225q"] Nov 01 15:54:20.151803 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.151786 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.166350 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.166322 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-646677ff99-h225q"] Nov 01 15:54:20.195812 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.195780 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnzv\" (UniqueName: \"kubernetes.io/projected/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-kube-api-access-7vnzv\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.195973 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.195828 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-webhook-cert\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.195973 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.195872 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-metrics-certs\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.297196 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.297163 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnzv\" (UniqueName: \"kubernetes.io/projected/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-kube-api-access-7vnzv\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.297367 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.297212 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-webhook-cert\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.297367 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.297279 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-metrics-certs\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.299939 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.299895 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-webhook-cert\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.300093 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.299989 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-metrics-certs\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.305204 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.305176 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnzv\" (UniqueName: \"kubernetes.io/projected/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-kube-api-access-7vnzv\") pod \"maas-controller-646677ff99-h225q\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.425987 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.425960 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-njj26"] Nov 01 15:54:20.428216 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:54:20.428188 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf367cd26_cc47_4cdc_96f4_e84a155546c9.slice/crio-ca400d7c6d9d9ec1aab4781bcb5aa7e351a74832635c8e3527898d2292139065 WatchSource:0}: Error finding container ca400d7c6d9d9ec1aab4781bcb5aa7e351a74832635c8e3527898d2292139065: Status 404 returned error can't find the container with id ca400d7c6d9d9ec1aab4781bcb5aa7e351a74832635c8e3527898d2292139065 Nov 01 15:54:20.490187 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.490148 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:20.771346 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.771321 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-646677ff99-h225q"] Nov 01 15:54:20.773405 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:54:20.773370 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2ea6f2_d2a7_41eb_8bd4_1993ecc75755.slice/crio-43e374c31fae90bc4480ec2f79d54a291dbf505a71f7f481ef744e8259d77fc3 WatchSource:0}: Error finding container 43e374c31fae90bc4480ec2f79d54a291dbf505a71f7f481ef744e8259d77fc3: Status 404 returned error can't find the container with id 43e374c31fae90bc4480ec2f79d54a291dbf505a71f7f481ef744e8259d77fc3 Nov 01 15:54:20.968404 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.968328 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-h225q" event={"ID":"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755","Type":"ContainerStarted","Data":"43e374c31fae90bc4480ec2f79d54a291dbf505a71f7f481ef744e8259d77fc3"} Nov 01 15:54:20.969456 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:20.969433 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-njj26" event={"ID":"f367cd26-cc47-4cdc-96f4-e84a155546c9","Type":"ContainerStarted","Data":"ca400d7c6d9d9ec1aab4781bcb5aa7e351a74832635c8e3527898d2292139065"} Nov 01 15:54:23.990769 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:23.990728 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-h225q" event={"ID":"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755","Type":"ContainerStarted","Data":"34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8"} Nov 01 15:54:23.991238 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:23.990893 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:23.992569 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:23.992546 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-njj26" event={"ID":"f367cd26-cc47-4cdc-96f4-e84a155546c9","Type":"ContainerStarted","Data":"e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08"} Nov 01 15:54:23.992699 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:23.992683 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:24.011928 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:24.011884 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-646677ff99-h225q" podStartSLOduration=1.158604068 podStartE2EDuration="4.011874692s" totalImagesPullingTime="2.853270624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:24.010296791 +0000 UTC m=+404.154751443" watchObservedRunningTime="2026-11-01 15:54:24.011874692 +0000 UTC m=+404.156329334" Nov 01 15:54:24.029159 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:24.029118 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-njj26" podStartSLOduration=1.976269976 podStartE2EDuration="5.029105874s" totalImagesPullingTime="3.052835898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:24.028478098 +0000 UTC m=+404.172932741" watchObservedRunningTime="2026-11-01 15:54:24.029105874 +0000 UTC m=+404.173560514" Nov 01 15:54:25.635707 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.635651 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2"] Nov 01 15:54:25.639640 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.639617 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.648654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.648625 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 15:54:25.648794 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.648771 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-spfdn\"" Nov 01 15:54:25.648864 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.648840 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 15:54:25.648959 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.648941 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 15:54:25.651651 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.651622 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2"] Nov 01 15:54:25.739097 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.739035 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/07a610d5-7012-466d-9d51-ff68f7119b52-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.739280 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.739110 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07a610d5-7012-466d-9d51-ff68f7119b52-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.739280 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.739147 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07a610d5-7012-466d-9d51-ff68f7119b52-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.739395 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.739311 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd577\" (UniqueName: \"kubernetes.io/projected/07a610d5-7012-466d-9d51-ff68f7119b52-kube-api-access-nd577\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.839741 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.839699 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd577\" (UniqueName: \"kubernetes.io/projected/07a610d5-7012-466d-9d51-ff68f7119b52-kube-api-access-nd577\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.839947 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.839787 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/07a610d5-7012-466d-9d51-ff68f7119b52-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.839947 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.839811 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07a610d5-7012-466d-9d51-ff68f7119b52-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.839947 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.839848 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07a610d5-7012-466d-9d51-ff68f7119b52-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.840686 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.840654 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07a610d5-7012-466d-9d51-ff68f7119b52-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.840875 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.840858 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/07a610d5-7012-466d-9d51-ff68f7119b52-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.842307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.842285 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/07a610d5-7012-466d-9d51-ff68f7119b52-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.847908 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.847887 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd577\" (UniqueName: \"kubernetes.io/projected/07a610d5-7012-466d-9d51-ff68f7119b52-kube-api-access-nd577\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hzdp2\" (UID: \"07a610d5-7012-466d-9d51-ff68f7119b52\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:25.960386 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:25.960291 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:26.246265 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:26.246234 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2"] Nov 01 15:54:26.248690 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:54:26.248650 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a610d5_7012_466d_9d51_ff68f7119b52.slice/crio-237ff24433f7a4733a88421e83184a643c593507876c0ba0b4e515b7ed16e710 WatchSource:0}: Error finding container 237ff24433f7a4733a88421e83184a643c593507876c0ba0b4e515b7ed16e710: Status 404 returned error can't find the container with id 237ff24433f7a4733a88421e83184a643c593507876c0ba0b4e515b7ed16e710 Nov 01 15:54:27.007125 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:27.007092 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" event={"ID":"07a610d5-7012-466d-9d51-ff68f7119b52","Type":"ContainerStarted","Data":"237ff24433f7a4733a88421e83184a643c593507876c0ba0b4e515b7ed16e710"} Nov 01 15:54:32.460587 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.457235 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dqb7v"] Nov 01 15:54:32.464580 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.464555 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.471901 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.471871 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dqb7v"] Nov 01 15:54:32.480420 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.480391 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2"] Nov 01 15:54:32.482584 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.482563 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 15:54:32.482692 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.482640 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wcc54\"" Nov 01 15:54:32.484288 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.484268 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.494705 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.494684 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mxp5w\"" Nov 01 15:54:32.498024 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.497998 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2"] Nov 01 15:54:32.503120 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.503094 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d3f317c-4ee8-4892-bbbb-ee45450abcb6-cert\") pod \"kserve-controller-manager-776cc66d47-dqb7v\" (UID: \"6d3f317c-4ee8-4892-bbbb-ee45450abcb6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.503263 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.503170 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9482ef96-d84e-4a5b-8716-e86b5d032213-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mkrl2\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.503311 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.503286 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rxz\" (UniqueName: \"kubernetes.io/projected/9482ef96-d84e-4a5b-8716-e86b5d032213-kube-api-access-p8rxz\") pod \"llmisvc-controller-manager-869d5dcc5d-mkrl2\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.503371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.503350 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l457c\" (UniqueName: \"kubernetes.io/projected/6d3f317c-4ee8-4892-bbbb-ee45450abcb6-kube-api-access-l457c\") pod \"kserve-controller-manager-776cc66d47-dqb7v\" (UID: \"6d3f317c-4ee8-4892-bbbb-ee45450abcb6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.511205 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.511168 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tqcrs"] Nov 01 15:54:32.515021 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.514987 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:32.529778 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.529751 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tqcrs"] Nov 01 15:54:32.582743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.582712 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 15:54:32.582912 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.582783 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-k9w8p\"" Nov 01 15:54:32.582912 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.582894 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 15:54:32.605712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.604179 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l457c\" (UniqueName: \"kubernetes.io/projected/6d3f317c-4ee8-4892-bbbb-ee45450abcb6-kube-api-access-l457c\") pod \"kserve-controller-manager-776cc66d47-dqb7v\" (UID: \"6d3f317c-4ee8-4892-bbbb-ee45450abcb6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.605712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.604271 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d3f317c-4ee8-4892-bbbb-ee45450abcb6-cert\") pod \"kserve-controller-manager-776cc66d47-dqb7v\" (UID: \"6d3f317c-4ee8-4892-bbbb-ee45450abcb6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.605712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.604319 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae4e15-1cb1-4c30-b15a-25b32cbd939b-tls-certs\") pod \"model-serving-api-75bd89fdfc-tqcrs\" (UID: \"9dae4e15-1cb1-4c30-b15a-25b32cbd939b\") " pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:32.605712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.604362 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9482ef96-d84e-4a5b-8716-e86b5d032213-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mkrl2\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.605712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.604410 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghznj\" (UniqueName: \"kubernetes.io/projected/9dae4e15-1cb1-4c30-b15a-25b32cbd939b-kube-api-access-ghznj\") pod \"model-serving-api-75bd89fdfc-tqcrs\" (UID: \"9dae4e15-1cb1-4c30-b15a-25b32cbd939b\") " pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:32.605712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.604436 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rxz\" (UniqueName: \"kubernetes.io/projected/9482ef96-d84e-4a5b-8716-e86b5d032213-kube-api-access-p8rxz\") pod \"llmisvc-controller-manager-869d5dcc5d-mkrl2\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.608619 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.608591 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d3f317c-4ee8-4892-bbbb-ee45450abcb6-cert\") pod \"kserve-controller-manager-776cc66d47-dqb7v\" (UID: \"6d3f317c-4ee8-4892-bbbb-ee45450abcb6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.615744 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.615718 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rxz\" (UniqueName: \"kubernetes.io/projected/9482ef96-d84e-4a5b-8716-e86b5d032213-kube-api-access-p8rxz\") pod \"llmisvc-controller-manager-869d5dcc5d-mkrl2\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.615892 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.615722 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l457c\" (UniqueName: \"kubernetes.io/projected/6d3f317c-4ee8-4892-bbbb-ee45450abcb6-kube-api-access-l457c\") pod \"kserve-controller-manager-776cc66d47-dqb7v\" (UID: \"6d3f317c-4ee8-4892-bbbb-ee45450abcb6\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.623567 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.623510 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9482ef96-d84e-4a5b-8716-e86b5d032213-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mkrl2\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.705613 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.705577 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae4e15-1cb1-4c30-b15a-25b32cbd939b-tls-certs\") pod \"model-serving-api-75bd89fdfc-tqcrs\" (UID: \"9dae4e15-1cb1-4c30-b15a-25b32cbd939b\") " pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:32.705803 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.705780 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghznj\" (UniqueName: \"kubernetes.io/projected/9dae4e15-1cb1-4c30-b15a-25b32cbd939b-kube-api-access-ghznj\") pod \"model-serving-api-75bd89fdfc-tqcrs\" (UID: \"9dae4e15-1cb1-4c30-b15a-25b32cbd939b\") " pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:32.708499 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.708475 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae4e15-1cb1-4c30-b15a-25b32cbd939b-tls-certs\") pod \"model-serving-api-75bd89fdfc-tqcrs\" (UID: \"9dae4e15-1cb1-4c30-b15a-25b32cbd939b\") " pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:32.719471 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.719393 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghznj\" (UniqueName: \"kubernetes.io/projected/9dae4e15-1cb1-4c30-b15a-25b32cbd939b-kube-api-access-ghznj\") pod \"model-serving-api-75bd89fdfc-tqcrs\" (UID: \"9dae4e15-1cb1-4c30-b15a-25b32cbd939b\") " pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:32.782240 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.782193 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:32.805071 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.804223 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:32.836105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:32.833582 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:33.090433 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:33.090397 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dqb7v"] Nov 01 15:54:33.090600 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:54:33.090574 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3f317c_4ee8_4892_bbbb_ee45450abcb6.slice/crio-523451650d18a78e48db82ccd7a855d135b7cd8d30617dbd00a4c41eb043a011 WatchSource:0}: Error finding container 523451650d18a78e48db82ccd7a855d135b7cd8d30617dbd00a4c41eb043a011: Status 404 returned error can't find the container with id 523451650d18a78e48db82ccd7a855d135b7cd8d30617dbd00a4c41eb043a011 Nov 01 15:54:33.119582 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:33.119555 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2"] Nov 01 15:54:33.120232 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:54:33.120203 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9482ef96_d84e_4a5b_8716_e86b5d032213.slice/crio-692e4fa30b00465cf07cc6aab9fa23b31ff2f67c287ede2b30d61caddd7f4451 WatchSource:0}: Error finding container 692e4fa30b00465cf07cc6aab9fa23b31ff2f67c287ede2b30d61caddd7f4451: Status 404 returned error can't find the container with id 692e4fa30b00465cf07cc6aab9fa23b31ff2f67c287ede2b30d61caddd7f4451 Nov 01 15:54:33.181669 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:33.181645 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tqcrs"] Nov 01 15:54:33.183903 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:54:33.183875 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dae4e15_1cb1_4c30_b15a_25b32cbd939b.slice/crio-c8327a8000cddb366a7711e65f2489223b93920e219feb25943a93e0de08381d WatchSource:0}: Error finding container c8327a8000cddb366a7711e65f2489223b93920e219feb25943a93e0de08381d: Status 404 returned error can't find the container with id c8327a8000cddb366a7711e65f2489223b93920e219feb25943a93e0de08381d Nov 01 15:54:34.060142 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:34.060050 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" event={"ID":"9dae4e15-1cb1-4c30-b15a-25b32cbd939b","Type":"ContainerStarted","Data":"c8327a8000cddb366a7711e65f2489223b93920e219feb25943a93e0de08381d"} Nov 01 15:54:34.062970 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:34.062884 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" event={"ID":"9482ef96-d84e-4a5b-8716-e86b5d032213","Type":"ContainerStarted","Data":"692e4fa30b00465cf07cc6aab9fa23b31ff2f67c287ede2b30d61caddd7f4451"} Nov 01 15:54:34.067443 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:34.067359 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" event={"ID":"6d3f317c-4ee8-4892-bbbb-ee45450abcb6","Type":"ContainerStarted","Data":"523451650d18a78e48db82ccd7a855d135b7cd8d30617dbd00a4c41eb043a011"} Nov 01 15:54:35.001022 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.000986 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:54:35.001242 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.001095 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:35.053491 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.053451 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-njj26"] Nov 01 15:54:35.107453 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.106680 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-njj26" podUID="f367cd26-cc47-4cdc-96f4-e84a155546c9" containerName="manager" containerID="cri-o://e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08" gracePeriod=30 Nov 01 15:54:35.447840 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.447799 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:35.543230 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.543143 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f367cd26-cc47-4cdc-96f4-e84a155546c9-kube-api-access-tcqhl\" (UniqueName: \"kubernetes.io/projected/f367cd26-cc47-4cdc-96f4-e84a155546c9-kube-api-access-tcqhl\") pod \"f367cd26-cc47-4cdc-96f4-e84a155546c9\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " Nov 01 15:54:35.543408 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.543314 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-webhook-cert\") pod \"f367cd26-cc47-4cdc-96f4-e84a155546c9\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " Nov 01 15:54:35.543408 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.543349 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-metrics-certs\") pod \"f367cd26-cc47-4cdc-96f4-e84a155546c9\" (UID: \"f367cd26-cc47-4cdc-96f4-e84a155546c9\") " Nov 01 15:54:35.547102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.547029 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f367cd26-cc47-4cdc-96f4-e84a155546c9-kube-api-access-tcqhl" pod "f367cd26-cc47-4cdc-96f4-e84a155546c9" (UID: "f367cd26-cc47-4cdc-96f4-e84a155546c9"). InnerVolumeSpecName "kube-api-access-tcqhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:54:35.547367 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.547315 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-metrics-certs" pod "f367cd26-cc47-4cdc-96f4-e84a155546c9" (UID: "f367cd26-cc47-4cdc-96f4-e84a155546c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:54:35.549359 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.549334 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-webhook-cert" pod "f367cd26-cc47-4cdc-96f4-e84a155546c9" (UID: "f367cd26-cc47-4cdc-96f4-e84a155546c9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:54:35.645569 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.644525 4911 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-webhook-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:35.645569 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.644563 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f367cd26-cc47-4cdc-96f4-e84a155546c9-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:35.645569 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:35.644580 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqhl\" (UniqueName: \"kubernetes.io/projected/f367cd26-cc47-4cdc-96f4-e84a155546c9-kube-api-access-tcqhl\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:54:36.117678 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.117638 4911 generic.go:356] "Generic (PLEG): container finished" podID="f367cd26-cc47-4cdc-96f4-e84a155546c9" containerID="e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08" exitCode=0 Nov 01 15:54:36.118097 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.117733 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-njj26" event={"ID":"f367cd26-cc47-4cdc-96f4-e84a155546c9","Type":"ContainerDied","Data":"e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08"} Nov 01 15:54:36.118097 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.117784 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-njj26" event={"ID":"f367cd26-cc47-4cdc-96f4-e84a155546c9","Type":"ContainerDied","Data":"ca400d7c6d9d9ec1aab4781bcb5aa7e351a74832635c8e3527898d2292139065"} Nov 01 15:54:36.118097 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.117813 4911 scope.go:122] "RemoveContainer" containerID="e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08" Nov 01 15:54:36.118097 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.117987 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-njj26" Nov 01 15:54:36.133423 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.133401 4911 scope.go:122] "RemoveContainer" containerID="e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08" Nov 01 15:54:36.133849 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:54:36.133817 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08\": container with ID starting with e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08 not found: ID does not exist" containerID="e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08" Nov 01 15:54:36.133955 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.133858 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08"} err="failed to get container status \"e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08\": rpc error: code = NotFound desc = could not find container \"e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08\": container with ID starting with e9f55c02008ad31d0c400bb24ee9dbebd91b5479804285ca64c51c4aded7bb08 not found: ID does not exist" Nov 01 15:54:36.137632 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.137579 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-njj26"] Nov 01 15:54:36.143368 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:36.143323 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-njj26"] Nov 01 15:54:37.126113 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:37.126076 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" event={"ID":"9dae4e15-1cb1-4c30-b15a-25b32cbd939b","Type":"ContainerStarted","Data":"2e52fa42936170c5e6dacec57e651f318e7e36e0e4d777acfa589581a41c4397"} Nov 01 15:54:37.126597 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:37.126165 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:37.148844 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:37.148786 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" podStartSLOduration=1.8904440340000002 podStartE2EDuration="5.148771931s" totalImagesPullingTime="3.258327897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:37.145591666 +0000 UTC m=+417.290046353" watchObservedRunningTime="2026-11-01 15:54:37.148771931 +0000 UTC m=+417.293226608" Nov 01 15:54:37.967670 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:37.967630 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f367cd26-cc47-4cdc-96f4-e84a155546c9" path="/var/lib/kubelet/pods/f367cd26-cc47-4cdc-96f4-e84a155546c9/volumes" Nov 01 15:54:38.135182 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:38.135134 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" event={"ID":"9482ef96-d84e-4a5b-8716-e86b5d032213","Type":"ContainerStarted","Data":"f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0"} Nov 01 15:54:38.135655 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:38.135336 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:54:38.137338 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:38.137268 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" event={"ID":"6d3f317c-4ee8-4892-bbbb-ee45450abcb6","Type":"ContainerStarted","Data":"ddc4eb64ae58082ab7f60712fd787f3f965a070c63abbcf686459f5eff90dc69"} Nov 01 15:54:38.155160 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:38.155098 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" podStartSLOduration=2.118526942 podStartE2EDuration="6.155079899s" totalImagesPullingTime="4.036552957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:38.153264065 +0000 UTC m=+418.297718707" watchObservedRunningTime="2026-11-01 15:54:38.155079899 +0000 UTC m=+418.299534545" Nov 01 15:54:38.175961 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:38.175908 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" podStartSLOduration=1.781918519 podStartE2EDuration="6.175894748s" totalImagesPullingTime="4.393976229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:38.174361654 +0000 UTC m=+418.318816295" watchObservedRunningTime="2026-11-01 15:54:38.175894748 +0000 UTC m=+418.320349398" Nov 01 15:54:39.142747 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:39.142712 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:54:40.148747 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:40.148706 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" event={"ID":"07a610d5-7012-466d-9d51-ff68f7119b52","Type":"ContainerStarted","Data":"e3355221570ddfe9c61bee23274e5258eeda3502b85c37384860198a4f27d351"} Nov 01 15:54:40.149177 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:40.148950 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:40.166808 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:40.166758 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" podStartSLOduration=2.147786564 podStartE2EDuration="15.166743712s" totalImagesPullingTime="13.018957148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:40.164788851 +0000 UTC m=+420.309243496" watchObservedRunningTime="2026-11-01 15:54:40.166743712 +0000 UTC m=+420.311198358" Nov 01 15:54:40.865057 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:54:40.864993 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:54:48.141722 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:48.141689 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-tqcrs" Nov 01 15:54:51.160301 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:51.160265 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hzdp2" Nov 01 15:54:55.393895 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.393857 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl"] Nov 01 15:54:55.394311 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.394275 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f367cd26-cc47-4cdc-96f4-e84a155546c9" containerName="manager" Nov 01 15:54:55.394311 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.394287 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f367cd26-cc47-4cdc-96f4-e84a155546c9" containerName="manager" Nov 01 15:54:55.394385 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.394366 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f367cd26-cc47-4cdc-96f4-e84a155546c9" containerName="manager" Nov 01 15:54:55.400543 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.400525 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.410249 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.410222 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl"] Nov 01 15:54:55.413721 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.413690 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 15:54:55.413862 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.413724 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 15:54:55.413862 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.413726 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 15:54:55.414126 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.414105 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 15:54:55.414227 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.414215 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 15:54:55.532463 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.532422 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr968\" (UniqueName: \"kubernetes.io/projected/feae6c1e-f455-4a6f-b82d-c321cf828574-kube-api-access-fr968\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.532628 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.532493 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-maas-api-tls\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.532628 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.532558 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-metrics-certs\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.532742 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.532664 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae6c1e-f455-4a6f-b82d-c321cf828574-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.634164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.634122 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae6c1e-f455-4a6f-b82d-c321cf828574-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.634350 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.634189 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr968\" (UniqueName: \"kubernetes.io/projected/feae6c1e-f455-4a6f-b82d-c321cf828574-kube-api-access-fr968\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.634350 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.634256 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-maas-api-tls\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.634350 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.634314 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-metrics-certs\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.634845 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.634822 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae6c1e-f455-4a6f-b82d-c321cf828574-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.636855 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.636833 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-maas-api-tls\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.636986 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.636965 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-metrics-certs\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.647505 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.647447 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr968\" (UniqueName: \"kubernetes.io/projected/feae6c1e-f455-4a6f-b82d-c321cf828574-kube-api-access-fr968\") pod \"maas-api-6d669c7fcd-xbpfl\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.714781 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.714750 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:55.992013 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:55.991985 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl"] Nov 01 15:54:55.993701 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:54:55.993667 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeae6c1e_f455_4a6f_b82d_c321cf828574.slice/crio-e8bf2a527b8f09d788daa702a3a691ee572a93bce3297923661bbf42f3b957ec WatchSource:0}: Error finding container e8bf2a527b8f09d788daa702a3a691ee572a93bce3297923661bbf42f3b957ec: Status 404 returned error can't find the container with id e8bf2a527b8f09d788daa702a3a691ee572a93bce3297923661bbf42f3b957ec Nov 01 15:54:56.230594 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:56.230556 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" event={"ID":"feae6c1e-f455-4a6f-b82d-c321cf828574","Type":"ContainerStarted","Data":"e8bf2a527b8f09d788daa702a3a691ee572a93bce3297923661bbf42f3b957ec"} Nov 01 15:54:58.275875 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:58.275832 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" event={"ID":"feae6c1e-f455-4a6f-b82d-c321cf828574","Type":"ContainerStarted","Data":"9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c"} Nov 01 15:54:58.276434 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:58.275943 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:54:58.294495 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:54:58.294446 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" podStartSLOduration=1.36734905 podStartE2EDuration="3.294430119s" totalImagesPullingTime="1.927081069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:58.292704895 +0000 UTC m=+438.437159540" watchObservedRunningTime="2026-11-01 15:54:58.294430119 +0000 UTC m=+438.438884784" Nov 01 15:55:04.286274 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:04.286242 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:55:04.991805 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:04.991768 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xgv22"] Nov 01 15:55:05.096220 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.096176 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xgv22"] Nov 01 15:55:05.096387 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.096247 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xgv22"] Nov 01 15:55:05.096387 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.096265 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-mw7xv"] Nov 01 15:55:05.096387 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.096189 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xgv22" Nov 01 15:55:05.096690 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:55:05.096666 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-swjz9], unattached volumes=[], failed to process volumes=[kube-api-access-swjz9]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-xgv22" podUID="73978e27-a757-429c-9520-ae4c5e4ef86f" Nov 01 15:55:05.117814 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.117776 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-mw7xv"] Nov 01 15:55:05.117965 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.117912 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.121735 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.121710 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 15:55:05.146104 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.146069 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf22ca66-ad9b-4774-9984-cedb72650bea-tls-cert\") pod \"authorino-6889cbd8c4-mw7xv\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.146275 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.146138 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77wj\" (UniqueName: \"kubernetes.io/projected/cf22ca66-ad9b-4774-9984-cedb72650bea-kube-api-access-m77wj\") pod \"authorino-6889cbd8c4-mw7xv\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.146275 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.146240 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjz9\" (UniqueName: \"kubernetes.io/projected/73978e27-a757-429c-9520-ae4c5e4ef86f-kube-api-access-swjz9\") pod \"authorino-6f6944bcdd-xgv22\" (UID: \"73978e27-a757-429c-9520-ae4c5e4ef86f\") " pod="kuadrant-system/authorino-6f6944bcdd-xgv22" Nov 01 15:55:05.247710 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.247622 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf22ca66-ad9b-4774-9984-cedb72650bea-tls-cert\") pod \"authorino-6889cbd8c4-mw7xv\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.247710 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.247704 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77wj\" (UniqueName: \"kubernetes.io/projected/cf22ca66-ad9b-4774-9984-cedb72650bea-kube-api-access-m77wj\") pod \"authorino-6889cbd8c4-mw7xv\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.247955 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.247732 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjz9\" (UniqueName: \"kubernetes.io/projected/73978e27-a757-429c-9520-ae4c5e4ef86f-kube-api-access-swjz9\") pod \"authorino-6f6944bcdd-xgv22\" (UID: \"73978e27-a757-429c-9520-ae4c5e4ef86f\") " pod="kuadrant-system/authorino-6f6944bcdd-xgv22" Nov 01 15:55:05.250299 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.250271 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf22ca66-ad9b-4774-9984-cedb72650bea-tls-cert\") pod \"authorino-6889cbd8c4-mw7xv\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.256183 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.256153 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77wj\" (UniqueName: \"kubernetes.io/projected/cf22ca66-ad9b-4774-9984-cedb72650bea-kube-api-access-m77wj\") pod \"authorino-6889cbd8c4-mw7xv\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.256599 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.256572 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjz9\" (UniqueName: \"kubernetes.io/projected/73978e27-a757-429c-9520-ae4c5e4ef86f-kube-api-access-swjz9\") pod \"authorino-6f6944bcdd-xgv22\" (UID: \"73978e27-a757-429c-9520-ae4c5e4ef86f\") " pod="kuadrant-system/authorino-6f6944bcdd-xgv22" Nov 01 15:55:05.313386 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.313355 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xgv22" Nov 01 15:55:05.323361 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.323330 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-mw7xv"] Nov 01 15:55:05.323608 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.323591 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:05.343905 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.343871 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68686bd5cc-9gpdn"] Nov 01 15:55:05.370197 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.370127 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xgv22" Nov 01 15:55:05.404395 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.404364 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68686bd5cc-9gpdn"] Nov 01 15:55:05.404538 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.404456 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:05.450151 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.450118 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73978e27-a757-429c-9520-ae4c5e4ef86f-kube-api-access-swjz9\" (UniqueName: \"kubernetes.io/projected/73978e27-a757-429c-9520-ae4c5e4ef86f-kube-api-access-swjz9\") pod \"73978e27-a757-429c-9520-ae4c5e4ef86f\" (UID: \"73978e27-a757-429c-9520-ae4c5e4ef86f\") " Nov 01 15:55:05.450416 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.450396 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-tls-cert\") pod \"authorino-68686bd5cc-9gpdn\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:05.450540 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.450520 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbpt\" (UniqueName: \"kubernetes.io/projected/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-kube-api-access-mcbpt\") pod \"authorino-68686bd5cc-9gpdn\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:05.452382 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.452360 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73978e27-a757-429c-9520-ae4c5e4ef86f-kube-api-access-swjz9" pod "73978e27-a757-429c-9520-ae4c5e4ef86f" (UID: "73978e27-a757-429c-9520-ae4c5e4ef86f"). InnerVolumeSpecName "kube-api-access-swjz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:55:05.551454 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.551362 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbpt\" (UniqueName: \"kubernetes.io/projected/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-kube-api-access-mcbpt\") pod \"authorino-68686bd5cc-9gpdn\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:05.551623 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.551466 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-tls-cert\") pod \"authorino-68686bd5cc-9gpdn\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:05.551623 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.551540 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjz9\" (UniqueName: \"kubernetes.io/projected/73978e27-a757-429c-9520-ae4c5e4ef86f-kube-api-access-swjz9\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:05.553969 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.553936 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-tls-cert\") pod \"authorino-68686bd5cc-9gpdn\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:05.560149 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.560128 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbpt\" (UniqueName: \"kubernetes.io/projected/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-kube-api-access-mcbpt\") pod \"authorino-68686bd5cc-9gpdn\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:05.686892 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.686860 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-mw7xv"] Nov 01 15:55:05.687442 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:55:05.687402 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf22ca66_ad9b_4774_9984_cedb72650bea.slice/crio-875ee935e514b54a975d9f5b578c472e88a064f686f92de86c2217e539220cf6 WatchSource:0}: Error finding container 875ee935e514b54a975d9f5b578c472e88a064f686f92de86c2217e539220cf6: Status 404 returned error can't find the container with id 875ee935e514b54a975d9f5b578c472e88a064f686f92de86c2217e539220cf6 Nov 01 15:55:05.783614 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:05.783582 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:55:06.067591 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.067560 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68686bd5cc-9gpdn"] Nov 01 15:55:06.069194 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:55:06.069158 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7bb12ed_4d57_423f_8fd5_549a5bb050f3.slice/crio-516d3ea23f74e1888a640944a93eed36073ce064c30bf1f8975e5de7873c10fd WatchSource:0}: Error finding container 516d3ea23f74e1888a640944a93eed36073ce064c30bf1f8975e5de7873c10fd: Status 404 returned error can't find the container with id 516d3ea23f74e1888a640944a93eed36073ce064c30bf1f8975e5de7873c10fd Nov 01 15:55:06.320084 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.320022 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" event={"ID":"cf22ca66-ad9b-4774-9984-cedb72650bea","Type":"ContainerStarted","Data":"875ee935e514b54a975d9f5b578c472e88a064f686f92de86c2217e539220cf6"} Nov 01 15:55:06.321270 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.321241 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" event={"ID":"f7bb12ed-4d57-423f-8fd5-549a5bb050f3","Type":"ContainerStarted","Data":"516d3ea23f74e1888a640944a93eed36073ce064c30bf1f8975e5de7873c10fd"} Nov 01 15:55:06.322803 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.322777 4911 generic.go:356] "Generic (PLEG): container finished" podID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerID="865f013190fc087ac222d9a18d64d7dc11c9f641147e95c7a9e240fedb8c80ba" exitCode=0 Nov 01 15:55:06.322912 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.322844 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" event={"ID":"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8","Type":"ContainerDied","Data":"865f013190fc087ac222d9a18d64d7dc11c9f641147e95c7a9e240fedb8c80ba"} Nov 01 15:55:06.322971 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.322951 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xgv22" Nov 01 15:55:06.323237 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.323221 4911 scope.go:122] "RemoveContainer" containerID="865f013190fc087ac222d9a18d64d7dc11c9f641147e95c7a9e240fedb8c80ba" Nov 01 15:55:06.324329 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.324294 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:55:06.324452 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.324365 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:55:06.324452 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.324397 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:55:06.364311 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.364277 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xgv22"] Nov 01 15:55:06.369405 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:06.369372 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xgv22"] Nov 01 15:55:07.332331 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.331754 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" event={"ID":"f7bb12ed-4d57-423f-8fd5-549a5bb050f3","Type":"ContainerStarted","Data":"a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294"} Nov 01 15:55:07.336261 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.336016 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" event={"ID":"f37e3bdc-1a02-41cb-8c76-61d11cbd86b8","Type":"ContainerStarted","Data":"daa3c2f7c76d1b3b6ab92803084ef68220eea796502a84f6de62da643114d1b9"} Nov 01 15:55:07.339329 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.338919 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" event={"ID":"cf22ca66-ad9b-4774-9984-cedb72650bea","Type":"ContainerStarted","Data":"6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761"} Nov 01 15:55:07.339329 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.339033 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" podUID="cf22ca66-ad9b-4774-9984-cedb72650bea" containerName="authorino" containerID="cri-o://6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761" gracePeriod=30 Nov 01 15:55:07.353095 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.349499 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" podStartSLOduration=1.594950135 podStartE2EDuration="2.349481492s" totalImagesPullingTime="754.531357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:55:07.347903156 +0000 UTC m=+447.492357798" watchObservedRunningTime="2026-11-01 15:55:07.349481492 +0000 UTC m=+447.493936133" Nov 01 15:55:07.393023 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.392648 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" podStartSLOduration=1.239223264 podStartE2EDuration="2.392632771s" totalImagesPullingTime="1.153409507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:55:07.370355835 +0000 UTC m=+447.514810477" watchObservedRunningTime="2026-11-01 15:55:07.392632771 +0000 UTC m=+447.537087411" Nov 01 15:55:07.396632 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.396593 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8zcbq"] Nov 01 15:55:07.396870 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.396821 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" podUID="b74b4de9-2b48-4cda-b92b-11ceaa4bbda5" containerName="authorino" containerID="cri-o://ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd" gracePeriod=30 Nov 01 15:55:07.781082 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.781017 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:07.792833 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.792795 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf22ca66-ad9b-4774-9984-cedb72650bea-tls-cert\" (UniqueName: \"kubernetes.io/secret/cf22ca66-ad9b-4774-9984-cedb72650bea-tls-cert\") pod \"cf22ca66-ad9b-4774-9984-cedb72650bea\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " Nov 01 15:55:07.792997 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.792892 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf22ca66-ad9b-4774-9984-cedb72650bea-kube-api-access-m77wj\" (UniqueName: \"kubernetes.io/projected/cf22ca66-ad9b-4774-9984-cedb72650bea-kube-api-access-m77wj\") pod \"cf22ca66-ad9b-4774-9984-cedb72650bea\" (UID: \"cf22ca66-ad9b-4774-9984-cedb72650bea\") " Nov 01 15:55:07.794984 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.794953 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf22ca66-ad9b-4774-9984-cedb72650bea-kube-api-access-m77wj" pod "cf22ca66-ad9b-4774-9984-cedb72650bea" (UID: "cf22ca66-ad9b-4774-9984-cedb72650bea"). InnerVolumeSpecName "kube-api-access-m77wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:55:07.797025 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.796999 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" Nov 01 15:55:07.804170 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.804145 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf22ca66-ad9b-4774-9984-cedb72650bea-tls-cert" pod "cf22ca66-ad9b-4774-9984-cedb72650bea" (UID: "cf22ca66-ad9b-4774-9984-cedb72650bea"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:55:07.894356 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.894269 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5-kube-api-access-tg8k4\" (UniqueName: \"kubernetes.io/projected/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5-kube-api-access-tg8k4\") pod \"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5\" (UID: \"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5\") " Nov 01 15:55:07.894531 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.894500 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m77wj\" (UniqueName: \"kubernetes.io/projected/cf22ca66-ad9b-4774-9984-cedb72650bea-kube-api-access-m77wj\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:07.894531 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.894514 4911 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf22ca66-ad9b-4774-9984-cedb72650bea-tls-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:07.896504 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.896474 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5-kube-api-access-tg8k4" pod "b74b4de9-2b48-4cda-b92b-11ceaa4bbda5" (UID: "b74b4de9-2b48-4cda-b92b-11ceaa4bbda5"). InnerVolumeSpecName "kube-api-access-tg8k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:55:07.980470 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.980432 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73978e27-a757-429c-9520-ae4c5e4ef86f" path="/var/lib/kubelet/pods/73978e27-a757-429c-9520-ae4c5e4ef86f/volumes" Nov 01 15:55:07.995397 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:07.995174 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8k4\" (UniqueName: \"kubernetes.io/projected/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5-kube-api-access-tg8k4\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:08.214766 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.214676 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:55:08.215984 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.215955 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9rb5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:55:08.216122 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.216015 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podUID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 01 15:55:08.345208 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.345177 4911 generic.go:356] "Generic (PLEG): container finished" podID="cf22ca66-ad9b-4774-9984-cedb72650bea" containerID="6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761" exitCode=0 Nov 01 15:55:08.345653 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.345223 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" Nov 01 15:55:08.345653 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.345271 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" event={"ID":"cf22ca66-ad9b-4774-9984-cedb72650bea","Type":"ContainerDied","Data":"6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761"} Nov 01 15:55:08.345653 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.345316 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-mw7xv" event={"ID":"cf22ca66-ad9b-4774-9984-cedb72650bea","Type":"ContainerDied","Data":"875ee935e514b54a975d9f5b578c472e88a064f686f92de86c2217e539220cf6"} Nov 01 15:55:08.345653 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.345335 4911 scope.go:122] "RemoveContainer" containerID="6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761" Nov 01 15:55:08.347032 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.347004 4911 generic.go:356] "Generic (PLEG): container finished" podID="b74b4de9-2b48-4cda-b92b-11ceaa4bbda5" containerID="ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd" exitCode=0 Nov 01 15:55:08.347176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.347090 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" Nov 01 15:55:08.347176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.347140 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" event={"ID":"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5","Type":"ContainerDied","Data":"ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd"} Nov 01 15:55:08.347294 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.347178 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8zcbq" event={"ID":"b74b4de9-2b48-4cda-b92b-11ceaa4bbda5","Type":"ContainerDied","Data":"7af0f7deddc7d9020a2d41e2974cb1aa548c19023d1cf720e63f2170bcd68455"} Nov 01 15:55:08.379972 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.379951 4911 scope.go:122] "RemoveContainer" containerID="6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761" Nov 01 15:55:08.380365 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:55:08.380342 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761\": container with ID starting with 6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761 not found: ID does not exist" containerID="6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761" Nov 01 15:55:08.380434 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.380375 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761"} err="failed to get container status \"6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761\": rpc error: code = NotFound desc = could not find container \"6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761\": container with ID starting with 6597f17351eb0b89338bd8aaefd5e9a641a3b1a03620803c67086551abcc5761 not found: ID does not exist" Nov 01 15:55:08.380434 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.380394 4911 scope.go:122] "RemoveContainer" containerID="ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd" Nov 01 15:55:08.385680 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.385659 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-mw7xv"] Nov 01 15:55:08.388993 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.388969 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-mw7xv"] Nov 01 15:55:08.389703 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.389686 4911 scope.go:122] "RemoveContainer" containerID="ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd" Nov 01 15:55:08.389967 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:55:08.389947 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd\": container with ID starting with ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd not found: ID does not exist" containerID="ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd" Nov 01 15:55:08.390029 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.389978 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd"} err="failed to get container status \"ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd\": rpc error: code = NotFound desc = could not find container \"ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd\": container with ID starting with ad2b4611eba4984a28e785a36e892cfbd92dbaddd501f8cec324bdec8188c3fd not found: ID does not exist" Nov 01 15:55:08.402262 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.402237 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8zcbq"] Nov 01 15:55:08.405560 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:08.405540 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8zcbq"] Nov 01 15:55:09.145512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:09.145478 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:55:09.215171 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:09.215132 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9rb5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:55:09.215328 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:09.215199 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podUID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 01 15:55:09.978854 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:09.978817 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74b4de9-2b48-4cda-b92b-11ceaa4bbda5" path="/var/lib/kubelet/pods/b74b4de9-2b48-4cda-b92b-11ceaa4bbda5/volumes" Nov 01 15:55:09.979242 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:09.979205 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf22ca66-ad9b-4774-9984-cedb72650bea" path="/var/lib/kubelet/pods/cf22ca66-ad9b-4774-9984-cedb72650bea/volumes" Nov 01 15:55:10.161339 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:10.161307 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dqb7v" Nov 01 15:55:10.214963 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:10.214929 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9rb5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:55:10.215182 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:10.214981 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podUID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 01 15:55:10.885829 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:55:10.885775 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:55:11.214598 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:11.214511 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9rb5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:55:11.214598 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:11.214569 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podUID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 01 15:55:12.214810 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:12.214770 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9rb5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 01 15:55:12.215239 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:12.214840 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podUID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 01 15:55:12.309308 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:12.309267 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:55:13.218813 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:13.218780 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:55:13.219801 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:13.219784 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" Nov 01 15:55:14.576484 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:14.576448 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl"] Nov 01 15:55:14.576924 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:14.576710 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" podUID="feae6c1e-f455-4a6f-b82d-c321cf828574" containerName="maas-api" containerID="cri-o://9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c" gracePeriod=30 Nov 01 15:55:15.403166 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.403131 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:55:15.409231 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.409201 4911 generic.go:356] "Generic (PLEG): container finished" podID="feae6c1e-f455-4a6f-b82d-c321cf828574" containerID="9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c" exitCode=0 Nov 01 15:55:15.409382 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.409237 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" Nov 01 15:55:15.409382 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.409285 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" event={"ID":"feae6c1e-f455-4a6f-b82d-c321cf828574","Type":"ContainerDied","Data":"9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c"} Nov 01 15:55:15.409382 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.409340 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl" event={"ID":"feae6c1e-f455-4a6f-b82d-c321cf828574","Type":"ContainerDied","Data":"e8bf2a527b8f09d788daa702a3a691ee572a93bce3297923661bbf42f3b957ec"} Nov 01 15:55:15.409382 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.409365 4911 scope.go:122] "RemoveContainer" containerID="9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c" Nov 01 15:55:15.421107 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.421086 4911 scope.go:122] "RemoveContainer" containerID="9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c" Nov 01 15:55:15.421449 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:55:15.421427 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c\": container with ID starting with 9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c not found: ID does not exist" containerID="9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c" Nov 01 15:55:15.421513 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.421461 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c"} err="failed to get container status \"9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c\": rpc error: code = NotFound desc = could not find container \"9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c\": container with ID starting with 9d18c837ae0f7df9189ed0ff05a306e0084e70ea5570d330dd4f6ecd5a98793c not found: ID does not exist" Nov 01 15:55:15.463501 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.463466 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-metrics-certs\") pod \"feae6c1e-f455-4a6f-b82d-c321cf828574\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " Nov 01 15:55:15.463678 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.463530 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feae6c1e-f455-4a6f-b82d-c321cf828574-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae6c1e-f455-4a6f-b82d-c321cf828574-openshift-service-ca\") pod \"feae6c1e-f455-4a6f-b82d-c321cf828574\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " Nov 01 15:55:15.463678 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.463562 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feae6c1e-f455-4a6f-b82d-c321cf828574-kube-api-access-fr968\" (UniqueName: \"kubernetes.io/projected/feae6c1e-f455-4a6f-b82d-c321cf828574-kube-api-access-fr968\") pod \"feae6c1e-f455-4a6f-b82d-c321cf828574\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " Nov 01 15:55:15.463678 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.463587 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-maas-api-tls\") pod \"feae6c1e-f455-4a6f-b82d-c321cf828574\" (UID: \"feae6c1e-f455-4a6f-b82d-c321cf828574\") " Nov 01 15:55:15.463962 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.463912 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feae6c1e-f455-4a6f-b82d-c321cf828574-openshift-service-ca" pod "feae6c1e-f455-4a6f-b82d-c321cf828574" (UID: "feae6c1e-f455-4a6f-b82d-c321cf828574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:55:15.465964 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.465937 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-maas-api-tls" pod "feae6c1e-f455-4a6f-b82d-c321cf828574" (UID: "feae6c1e-f455-4a6f-b82d-c321cf828574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:55:15.466102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.466061 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feae6c1e-f455-4a6f-b82d-c321cf828574-kube-api-access-fr968" pod "feae6c1e-f455-4a6f-b82d-c321cf828574" (UID: "feae6c1e-f455-4a6f-b82d-c321cf828574"). InnerVolumeSpecName "kube-api-access-fr968". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:55:15.466154 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.466133 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-metrics-certs" pod "feae6c1e-f455-4a6f-b82d-c321cf828574" (UID: "feae6c1e-f455-4a6f-b82d-c321cf828574"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:55:15.565134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.565033 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:15.565134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.565084 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/feae6c1e-f455-4a6f-b82d-c321cf828574-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:15.565134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.565095 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr968\" (UniqueName: \"kubernetes.io/projected/feae6c1e-f455-4a6f-b82d-c321cf828574-kube-api-access-fr968\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:15.565134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.565107 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/feae6c1e-f455-4a6f-b82d-c321cf828574-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:55:15.732665 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.732627 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl"] Nov 01 15:55:15.734950 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.734919 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xbpfl"] Nov 01 15:55:15.971331 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:15.971237 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feae6c1e-f455-4a6f-b82d-c321cf828574" path="/var/lib/kubelet/pods/feae6c1e-f455-4a6f-b82d-c321cf828574/volumes" Nov 01 15:55:28.424071 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424026 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k"] Nov 01 15:55:28.424512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424461 4911 cpu_manager.go:414] "containerMap: removing container" podUID="cf22ca66-ad9b-4774-9984-cedb72650bea" containerName="authorino" Nov 01 15:55:28.424512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424476 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf22ca66-ad9b-4774-9984-cedb72650bea" containerName="authorino" Nov 01 15:55:28.424512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424492 4911 cpu_manager.go:414] "containerMap: removing container" podUID="feae6c1e-f455-4a6f-b82d-c321cf828574" containerName="maas-api" Nov 01 15:55:28.424512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424498 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feae6c1e-f455-4a6f-b82d-c321cf828574" containerName="maas-api" Nov 01 15:55:28.424512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424510 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b74b4de9-2b48-4cda-b92b-11ceaa4bbda5" containerName="authorino" Nov 01 15:55:28.424512 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424515 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74b4de9-2b48-4cda-b92b-11ceaa4bbda5" containerName="authorino" Nov 01 15:55:28.424778 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424591 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf22ca66-ad9b-4774-9984-cedb72650bea" containerName="authorino" Nov 01 15:55:28.424778 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424604 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="feae6c1e-f455-4a6f-b82d-c321cf828574" containerName="maas-api" Nov 01 15:55:28.424778 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.424613 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74b4de9-2b48-4cda-b92b-11ceaa4bbda5" containerName="authorino" Nov 01 15:55:28.428015 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.427991 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:28.440219 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.440195 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k"] Nov 01 15:55:28.481196 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.481169 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44qn4\" (UniqueName: \"kubernetes.io/projected/62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4-kube-api-access-44qn4\") pod \"llmisvc-controller-manager-6466c554d8-sbw8k\" (UID: \"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:28.481354 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.481209 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4-cert\") pod \"llmisvc-controller-manager-6466c554d8-sbw8k\" (UID: \"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:28.582749 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.582714 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44qn4\" (UniqueName: \"kubernetes.io/projected/62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4-kube-api-access-44qn4\") pod \"llmisvc-controller-manager-6466c554d8-sbw8k\" (UID: \"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:28.582925 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.582761 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4-cert\") pod \"llmisvc-controller-manager-6466c554d8-sbw8k\" (UID: \"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:28.585185 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.585165 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4-cert\") pod \"llmisvc-controller-manager-6466c554d8-sbw8k\" (UID: \"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:28.591328 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.591307 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44qn4\" (UniqueName: \"kubernetes.io/projected/62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4-kube-api-access-44qn4\") pod \"llmisvc-controller-manager-6466c554d8-sbw8k\" (UID: \"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:28.741330 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:28.741299 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:29.012208 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:29.012180 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k"] Nov 01 15:55:29.014197 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:55:29.014168 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ad4809_9d4b_4a4c_8ea2_26cf5cb427b4.slice/crio-392e277a228c9e1ccb5a1f25ed9a02c5d447a73348cd8158d4003337da2afc87 WatchSource:0}: Error finding container 392e277a228c9e1ccb5a1f25ed9a02c5d447a73348cd8158d4003337da2afc87: Status 404 returned error can't find the container with id 392e277a228c9e1ccb5a1f25ed9a02c5d447a73348cd8158d4003337da2afc87 Nov 01 15:55:29.481333 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:29.481304 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" event={"ID":"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4","Type":"ContainerStarted","Data":"392e277a228c9e1ccb5a1f25ed9a02c5d447a73348cd8158d4003337da2afc87"} Nov 01 15:55:30.488764 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:30.488727 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" event={"ID":"62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4","Type":"ContainerStarted","Data":"92bd7b51e4797a417801d607f7ba965924b32c3944c8731da81caed212cadaae"} Nov 01 15:55:30.489183 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:30.488814 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:55:30.506299 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:55:30.506253 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" podStartSLOduration=2.09272695 podStartE2EDuration="2.506241041s" totalImagesPullingTime="413.514091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:55:30.504567466 +0000 UTC m=+470.649022106" watchObservedRunningTime="2026-11-01 15:55:30.506241041 +0000 UTC m=+470.650695681" Nov 01 15:55:40.864987 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:55:40.864950 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:56:01.497441 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.497363 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sbw8k" Nov 01 15:56:01.544348 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.544313 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2"] Nov 01 15:56:01.544550 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.544527 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" podUID="9482ef96-d84e-4a5b-8716-e86b5d032213" containerName="manager" containerID="cri-o://f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0" gracePeriod=30 Nov 01 15:56:01.794937 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.794911 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:56:01.858223 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.858194 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9482ef96-d84e-4a5b-8716-e86b5d032213-kube-api-access-p8rxz\" (UniqueName: \"kubernetes.io/projected/9482ef96-d84e-4a5b-8716-e86b5d032213-kube-api-access-p8rxz\") pod \"9482ef96-d84e-4a5b-8716-e86b5d032213\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " Nov 01 15:56:01.858380 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.858240 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9482ef96-d84e-4a5b-8716-e86b5d032213-cert\" (UniqueName: \"kubernetes.io/secret/9482ef96-d84e-4a5b-8716-e86b5d032213-cert\") pod \"9482ef96-d84e-4a5b-8716-e86b5d032213\" (UID: \"9482ef96-d84e-4a5b-8716-e86b5d032213\") " Nov 01 15:56:01.860332 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.860299 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9482ef96-d84e-4a5b-8716-e86b5d032213-kube-api-access-p8rxz" pod "9482ef96-d84e-4a5b-8716-e86b5d032213" (UID: "9482ef96-d84e-4a5b-8716-e86b5d032213"). InnerVolumeSpecName "kube-api-access-p8rxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:01.860450 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.860338 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9482ef96-d84e-4a5b-8716-e86b5d032213-cert" pod "9482ef96-d84e-4a5b-8716-e86b5d032213" (UID: "9482ef96-d84e-4a5b-8716-e86b5d032213"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:56:01.959639 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.959603 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rxz\" (UniqueName: \"kubernetes.io/projected/9482ef96-d84e-4a5b-8716-e86b5d032213-kube-api-access-p8rxz\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:01.959639 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:01.959637 4911 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9482ef96-d84e-4a5b-8716-e86b5d032213-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:02.648355 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.648322 4911 generic.go:356] "Generic (PLEG): container finished" podID="9482ef96-d84e-4a5b-8716-e86b5d032213" containerID="f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0" exitCode=0 Nov 01 15:56:02.648786 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.648364 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" Nov 01 15:56:02.648786 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.648405 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" event={"ID":"9482ef96-d84e-4a5b-8716-e86b5d032213","Type":"ContainerDied","Data":"f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0"} Nov 01 15:56:02.648786 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.648457 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2" event={"ID":"9482ef96-d84e-4a5b-8716-e86b5d032213","Type":"ContainerDied","Data":"692e4fa30b00465cf07cc6aab9fa23b31ff2f67c287ede2b30d61caddd7f4451"} Nov 01 15:56:02.648786 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.648481 4911 scope.go:122] "RemoveContainer" containerID="f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0" Nov 01 15:56:02.656958 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.656928 4911 scope.go:122] "RemoveContainer" containerID="f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0" Nov 01 15:56:02.657250 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:56:02.657228 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0\": container with ID starting with f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0 not found: ID does not exist" containerID="f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0" Nov 01 15:56:02.657376 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.657262 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0"} err="failed to get container status \"f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0\": rpc error: code = NotFound desc = could not find container \"f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0\": container with ID starting with f0726929d96f97c8509651e5b1541ea5a66f5447ec4f7163b470d6a749d3cfb0 not found: ID does not exist" Nov 01 15:56:02.664549 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.664526 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2"] Nov 01 15:56:02.671264 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:02.671243 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mkrl2"] Nov 01 15:56:03.965239 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:03.965196 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9482ef96-d84e-4a5b-8716-e86b5d032213" path="/var/lib/kubelet/pods/9482ef96-d84e-4a5b-8716-e86b5d032213/volumes" Nov 01 15:56:10.864531 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:56:10.864494 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:56:40.866665 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:56:40.866632 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:56:46.205378 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.205335 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl"] Nov 01 15:56:46.205988 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.205967 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9482ef96-d84e-4a5b-8716-e86b5d032213" containerName="manager" Nov 01 15:56:46.206061 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.205992 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9482ef96-d84e-4a5b-8716-e86b5d032213" containerName="manager" Nov 01 15:56:46.206124 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.206111 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9482ef96-d84e-4a5b-8716-e86b5d032213" containerName="manager" Nov 01 15:56:46.211071 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.211026 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.228030 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.228006 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl"] Nov 01 15:56:46.281583 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281545 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxlw\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-kube-api-access-8vxlw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281779 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281591 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281779 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281680 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281779 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281730 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281796 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281852 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281869 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281886 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.281919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.281910 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383340 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383306 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383340 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383343 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383543 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383361 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383543 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383415 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383614 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383561 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxlw\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-kube-api-access-8vxlw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383655 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383616 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383694 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383661 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383745 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383695 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383745 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383721 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383745 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383728 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.383897 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.383790 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.384063 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.384019 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.384208 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.384192 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.384404 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.384381 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.385687 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.385658 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.385959 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.385943 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.390607 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.390581 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.390814 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.390799 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxlw\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-kube-api-access-8vxlw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hfzvl\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.525092 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.525060 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:46.803707 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.803677 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl"] Nov 01 15:56:46.805880 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:56:46.805848 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c8d771_57c0_4eec_9062_c2cf60a3aa79.slice/crio-34f1bb32c66e077287d1788f41af09dc5d277afc960292f541dab47d30f0752c WatchSource:0}: Error finding container 34f1bb32c66e077287d1788f41af09dc5d277afc960292f541dab47d30f0752c: Status 404 returned error can't find the container with id 34f1bb32c66e077287d1788f41af09dc5d277afc960292f541dab47d30f0752c Nov 01 15:56:46.808011 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.807977 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:46.808131 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.808070 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:46.808131 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.808122 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:46.890434 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.890397 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" event={"ID":"68c8d771-57c0-4eec-9062-c2cf60a3aa79","Type":"ContainerStarted","Data":"0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d"} Nov 01 15:56:46.890613 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.890451 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" event={"ID":"68c8d771-57c0-4eec-9062-c2cf60a3aa79","Type":"ContainerStarted","Data":"34f1bb32c66e077287d1788f41af09dc5d277afc960292f541dab47d30f0752c"} Nov 01 15:56:46.908834 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:46.908786 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podStartSLOduration=0.908772832 podStartE2EDuration="908.772832ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:46.907785846 +0000 UTC m=+547.052240503" watchObservedRunningTime="2026-11-01 15:56:46.908772832 +0000 UTC m=+547.053227472" Nov 01 15:56:47.525950 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:47.525924 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:47.527552 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:47.527528 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 15:56:47.527664 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:47.527576 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 15:56:48.526334 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:48.526298 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 15:56:48.526711 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:48.526374 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 15:56:49.525498 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:49.525460 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 15:56:49.525679 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:49.525518 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 15:56:50.525782 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:50.525748 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 15:56:50.526200 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:50.525831 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 15:56:51.526010 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:51.525974 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 15:56:51.526433 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:51.526067 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 15:56:52.525506 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:52.525463 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 15:56:52.525695 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:52.525523 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 15:56:52.921409 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:52.921325 4911 generic.go:356] "Generic (PLEG): container finished" podID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerID="34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8" exitCode=1 Nov 01 15:56:52.921409 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:52.921376 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-h225q" event={"ID":"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755","Type":"ContainerDied","Data":"34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8"} Nov 01 15:56:52.921911 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:52.921749 4911 scope.go:122] "RemoveContainer" containerID="34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8" Nov 01 15:56:53.526004 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:53.525970 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 15:56:53.526202 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:53.526032 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 15:56:53.928262 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:53.928226 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-h225q" event={"ID":"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755","Type":"ContainerStarted","Data":"31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef"} Nov 01 15:56:53.928657 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:53.928466 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:56:55.012703 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:55.012672 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:55.013198 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:55.012969 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:56:55.013893 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:56:55.013875 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 15:57:01.234524 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.234487 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj"] Nov 01 15:57:01.238362 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.238339 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.253134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.253106 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj"] Nov 01 15:57:01.321238 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321197 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321238 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321236 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321451 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321259 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btjm2\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-kube-api-access-btjm2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321451 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321344 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321451 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321413 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321546 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321453 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321608 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321562 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321642 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321605 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.321642 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.321631 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955395d-d20b-4df9-ae0f-8da5715e29b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423007 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.422960 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423007 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423006 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423064 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btjm2\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-kube-api-access-btjm2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423096 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423120 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423146 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423231 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423261 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423301 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955395d-d20b-4df9-ae0f-8da5715e29b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423677 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423626 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423677 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423662 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423795 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423677 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.423903 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.423881 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.424068 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.424031 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955395d-d20b-4df9-ae0f-8da5715e29b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.425414 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.425394 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.425638 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.425617 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.431755 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.431724 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.431875 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.431847 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btjm2\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-kube-api-access-btjm2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-whjkj\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.553421 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.553382 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:01.832871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.832825 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj"] Nov 01 15:57:01.835835 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:57:01.835802 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3955395d_d20b_4df9_ae0f_8da5715e29b3.slice/crio-fcca20f5b04652b3d23d69a4d185d7d46f228d922d6225193cff9d501389dc59 WatchSource:0}: Error finding container fcca20f5b04652b3d23d69a4d185d7d46f228d922d6225193cff9d501389dc59: Status 404 returned error can't find the container with id fcca20f5b04652b3d23d69a4d185d7d46f228d922d6225193cff9d501389dc59 Nov 01 15:57:01.838174 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.838140 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.838271 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.838222 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.838271 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.838266 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.973419 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.973378 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" event={"ID":"3955395d-d20b-4df9-ae0f-8da5715e29b3","Type":"ContainerStarted","Data":"cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9"} Nov 01 15:57:01.973602 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.973445 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" event={"ID":"3955395d-d20b-4df9-ae0f-8da5715e29b3","Type":"ContainerStarted","Data":"fcca20f5b04652b3d23d69a4d185d7d46f228d922d6225193cff9d501389dc59"} Nov 01 15:57:01.994887 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:01.994812 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podStartSLOduration=0.994792257 podStartE2EDuration="994.792257ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:01.992516935 +0000 UTC m=+562.136971567" watchObservedRunningTime="2026-11-01 15:57:01.994792257 +0000 UTC m=+562.139246994" Nov 01 15:57:02.574085 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:02.567426 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:02.574625 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:02.574499 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-whjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:57:02.574625 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:02.574565 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 15:57:03.561524 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:03.561480 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-whjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:57:03.561777 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:03.561747 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 15:57:04.563082 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:04.562513 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-whjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:57:04.563082 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:04.562585 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 15:57:04.939723 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:04.939641 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 15:57:05.574201 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:05.574159 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-whjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:57:05.574654 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:05.574222 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 15:57:06.563486 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:06.563446 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-whjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:57:06.563810 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:06.563780 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 15:57:07.558480 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:07.558439 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-whjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:57:07.559222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:07.559189 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 15:57:08.554785 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:08.554750 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-whjkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:57:08.554962 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:08.554807 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 15:57:10.088363 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:10.088327 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:10.089096 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:10.089076 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:10.089915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:10.089890 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 15:57:10.865186 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:57:10.865155 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:57:12.617704 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.617665 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f5446d75d-76bmb"] Nov 01 15:57:12.621666 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.621641 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:12.627589 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.627564 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5446d75d-76bmb"] Nov 01 15:57:12.746276 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.746239 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ad863-8cfe-4c57-9250-f07bb5329feb-tls-cert\") pod \"authorino-6f5446d75d-76bmb\" (UID: \"233ad863-8cfe-4c57-9250-f07bb5329feb\") " pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:12.746441 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.746287 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xkg\" (UniqueName: \"kubernetes.io/projected/233ad863-8cfe-4c57-9250-f07bb5329feb-kube-api-access-n7xkg\") pod \"authorino-6f5446d75d-76bmb\" (UID: \"233ad863-8cfe-4c57-9250-f07bb5329feb\") " pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:12.846862 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.846822 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ad863-8cfe-4c57-9250-f07bb5329feb-tls-cert\") pod \"authorino-6f5446d75d-76bmb\" (UID: \"233ad863-8cfe-4c57-9250-f07bb5329feb\") " pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:12.847070 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.846877 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xkg\" (UniqueName: \"kubernetes.io/projected/233ad863-8cfe-4c57-9250-f07bb5329feb-kube-api-access-n7xkg\") pod \"authorino-6f5446d75d-76bmb\" (UID: \"233ad863-8cfe-4c57-9250-f07bb5329feb\") " pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:12.849380 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.849355 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/233ad863-8cfe-4c57-9250-f07bb5329feb-tls-cert\") pod \"authorino-6f5446d75d-76bmb\" (UID: \"233ad863-8cfe-4c57-9250-f07bb5329feb\") " pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:12.855374 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.855349 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xkg\" (UniqueName: \"kubernetes.io/projected/233ad863-8cfe-4c57-9250-f07bb5329feb-kube-api-access-n7xkg\") pod \"authorino-6f5446d75d-76bmb\" (UID: \"233ad863-8cfe-4c57-9250-f07bb5329feb\") " pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:12.935781 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:12.935691 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5446d75d-76bmb" Nov 01 15:57:13.207139 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:13.207097 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5446d75d-76bmb"] Nov 01 15:57:13.209379 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:57:13.209340 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233ad863_8cfe_4c57_9250_f07bb5329feb.slice/crio-867b30f39c07ad18f58c1fd8dc8dd5445c2f2a5862d489384aaad4ee861bf267 WatchSource:0}: Error finding container 867b30f39c07ad18f58c1fd8dc8dd5445c2f2a5862d489384aaad4ee861bf267: Status 404 returned error can't find the container with id 867b30f39c07ad18f58c1fd8dc8dd5445c2f2a5862d489384aaad4ee861bf267 Nov 01 15:57:14.043371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:14.043326 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5446d75d-76bmb" event={"ID":"233ad863-8cfe-4c57-9250-f07bb5329feb","Type":"ContainerStarted","Data":"867b30f39c07ad18f58c1fd8dc8dd5445c2f2a5862d489384aaad4ee861bf267"} Nov 01 15:57:15.058745 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:15.058702 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5446d75d-76bmb" event={"ID":"233ad863-8cfe-4c57-9250-f07bb5329feb","Type":"ContainerStarted","Data":"6f696ff858925918bbf7d7f1e35d855cfbbb3ccd7364949b6253e6ffeb5d2ae5"} Nov 01 15:57:15.099302 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:15.099243 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f5446d75d-76bmb" podStartSLOduration=1.978841063 podStartE2EDuration="3.099222935s" totalImagesPullingTime="1.120381872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:15.09544602 +0000 UTC m=+575.239900659" watchObservedRunningTime="2026-11-01 15:57:15.099222935 +0000 UTC m=+575.243677580" Nov 01 15:57:15.137691 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:15.137643 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68686bd5cc-9gpdn"] Nov 01 15:57:15.137900 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:15.137861 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" podUID="f7bb12ed-4d57-423f-8fd5-549a5bb050f3" containerName="authorino" containerID="cri-o://a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294" gracePeriod=30 Nov 01 15:57:15.971371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:15.971341 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:57:16.066404 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.066303 4911 generic.go:356] "Generic (PLEG): container finished" podID="f7bb12ed-4d57-423f-8fd5-549a5bb050f3" containerID="a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294" exitCode=0 Nov 01 15:57:16.066404 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.066351 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" Nov 01 15:57:16.066929 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.066390 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" event={"ID":"f7bb12ed-4d57-423f-8fd5-549a5bb050f3","Type":"ContainerDied","Data":"a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294"} Nov 01 15:57:16.066929 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.066445 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68686bd5cc-9gpdn" event={"ID":"f7bb12ed-4d57-423f-8fd5-549a5bb050f3","Type":"ContainerDied","Data":"516d3ea23f74e1888a640944a93eed36073ce064c30bf1f8975e5de7873c10fd"} Nov 01 15:57:16.066929 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.066468 4911 scope.go:122] "RemoveContainer" containerID="a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294" Nov 01 15:57:16.075840 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.075819 4911 scope.go:122] "RemoveContainer" containerID="a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294" Nov 01 15:57:16.076172 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:57:16.076143 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294\": container with ID starting with a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294 not found: ID does not exist" containerID="a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294" Nov 01 15:57:16.076301 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.076180 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294"} err="failed to get container status \"a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294\": rpc error: code = NotFound desc = could not find container \"a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294\": container with ID starting with a4cb417fde2d77d8d99e946ed8753b22b3dcec198400cead6e8728240b3a5294 not found: ID does not exist" Nov 01 15:57:16.122112 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.120295 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-tls-cert\" (UniqueName: \"kubernetes.io/secret/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-tls-cert\") pod \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " Nov 01 15:57:16.122112 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.120453 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-kube-api-access-mcbpt\" (UniqueName: \"kubernetes.io/projected/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-kube-api-access-mcbpt\") pod \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\" (UID: \"f7bb12ed-4d57-423f-8fd5-549a5bb050f3\") " Nov 01 15:57:16.129207 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.129166 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-kube-api-access-mcbpt" pod "f7bb12ed-4d57-423f-8fd5-549a5bb050f3" (UID: "f7bb12ed-4d57-423f-8fd5-549a5bb050f3"). InnerVolumeSpecName "kube-api-access-mcbpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:57:16.138675 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.138639 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-tls-cert" pod "f7bb12ed-4d57-423f-8fd5-549a5bb050f3" (UID: "f7bb12ed-4d57-423f-8fd5-549a5bb050f3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:57:16.222224 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.222186 4911 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-tls-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:57:16.222224 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.222220 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbpt\" (UniqueName: \"kubernetes.io/projected/f7bb12ed-4d57-423f-8fd5-549a5bb050f3-kube-api-access-mcbpt\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:57:16.428640 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.428586 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68686bd5cc-9gpdn"] Nov 01 15:57:16.430899 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:16.430872 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68686bd5cc-9gpdn"] Nov 01 15:57:17.965839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:17.965805 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bb12ed-4d57-423f-8fd5-549a5bb050f3" path="/var/lib/kubelet/pods/f7bb12ed-4d57-423f-8fd5-549a5bb050f3/volumes" Nov 01 15:57:40.115009 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:40.114980 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 15:57:40.122020 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:57:40.121998 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 15:57:40.865267 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:57:40.865224 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:58:10.865782 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:58:10.865745 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:58:13.278899 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.278865 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf"] Nov 01 15:58:13.279486 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.279272 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f7bb12ed-4d57-423f-8fd5-549a5bb050f3" containerName="authorino" Nov 01 15:58:13.279486 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.279284 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bb12ed-4d57-423f-8fd5-549a5bb050f3" containerName="authorino" Nov 01 15:58:13.279486 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.279376 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bb12ed-4d57-423f-8fd5-549a5bb050f3" containerName="authorino" Nov 01 15:58:13.282671 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.282651 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.294863 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.294815 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-164db6-openshift-default-dockercfg-mt9cn\"" Nov 01 15:58:13.309927 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.309899 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf"] Nov 01 15:58:13.421944 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.421907 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.421973 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c728783-d532-41ca-9ea3-b50340b395cd-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.422128 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422245 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.422203 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422245 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.422235 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422429 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.422405 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422462 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.422449 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgblt\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-kube-api-access-jgblt\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422504 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.422478 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.422573 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.422558 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c728783-d532-41ca-9ea3-b50340b395cd-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.523697 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523662 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.523886 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523703 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.523886 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523786 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.523886 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523813 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgblt\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-kube-api-access-jgblt\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.523886 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523839 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.523886 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523875 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c728783-d532-41ca-9ea3-b50340b395cd-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.524170 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523924 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.524170 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.523973 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c728783-d532-41ca-9ea3-b50340b395cd-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.524170 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.524002 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.524321 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.524241 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.524520 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.524496 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.524616 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.524577 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.524727 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.524698 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c728783-d532-41ca-9ea3-b50340b395cd-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.525034 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.524999 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.526885 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.526861 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.527568 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.527545 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c728783-d532-41ca-9ea3-b50340b395cd-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.568187 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.568115 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgblt\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-kube-api-access-jgblt\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.569659 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.569634 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.595555 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.595523 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:13.814267 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.814228 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx"] Nov 01 15:58:13.819579 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.819515 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.830733 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.829903 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6b0e0f-openshift-default-dockercfg-bgk88\"" Nov 01 15:58:13.833204 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.833179 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx"] Nov 01 15:58:13.890344 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.890317 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf"] Nov 01 15:58:13.892941 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:13.892909 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c728783_d532_41ca_9ea3_b50340b395cd.slice/crio-8dbb525730365a0b17b6e1744162d2fb692ee5c55873f2c20d9501a837d53766 WatchSource:0}: Error finding container 8dbb525730365a0b17b6e1744162d2fb692ee5c55873f2c20d9501a837d53766: Status 404 returned error can't find the container with id 8dbb525730365a0b17b6e1744162d2fb692ee5c55873f2c20d9501a837d53766 Nov 01 15:58:13.895824 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.895786 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:13.896030 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.896006 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:13.896233 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.896209 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:13.915397 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.915356 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh"] Nov 01 15:58:13.920825 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.920803 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:13.930444 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930420 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930558 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930467 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930597 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930551 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930597 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930588 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930709 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930682 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf66b\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-kube-api-access-mf66b\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930746 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930734 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930825 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930807 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930865 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930851 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.930913 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.930879 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:13.933466 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.933448 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bd831a-openshift-default-dockercfg-f9pt7\"" Nov 01 15:58:13.936805 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:13.936775 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh"] Nov 01 15:58:14.032008 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.031978 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032140 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032032 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b79f2c-6850-468e-8994-0c409eead207-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032140 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032095 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032237 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032178 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf66b\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-kube-api-access-mf66b\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032283 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032243 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032283 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032277 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032296 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032337 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032381 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032368 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032540 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032396 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032540 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032434 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032540 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032464 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032754 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032611 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwgf\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-kube-api-access-zmwgf\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032754 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032650 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032754 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032682 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.032912 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032760 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032912 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032764 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032912 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032851 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.032912 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032884 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b79f2c-6850-468e-8994-0c409eead207-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.033154 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032914 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.033154 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032930 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.033154 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.032949 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.033643 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.033617 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.035015 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.034993 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.035277 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.035256 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.080133 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.067651 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.080133 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.068333 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf66b\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-kube-api-access-mf66b\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.133952 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.133897 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134184 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134075 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b79f2c-6850-468e-8994-0c409eead207-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134184 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134128 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134184 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134168 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b79f2c-6850-468e-8994-0c409eead207-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134194 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134305 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134331 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134365 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134575 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134406 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwgf\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-kube-api-access-zmwgf\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134633 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134602 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.134907 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134883 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.135003 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.134973 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.135297 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.135274 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.135919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.135895 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b79f2c-6850-468e-8994-0c409eead207-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.137484 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.137420 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b79f2c-6850-468e-8994-0c409eead207-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.137586 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.137494 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.140670 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.140643 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:14.143082 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.143060 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.143773 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.143751 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwgf\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-kube-api-access-zmwgf\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.302400 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.302364 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:14.379688 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.379650 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" event={"ID":"4c728783-d532-41ca-9ea3-b50340b395cd","Type":"ContainerStarted","Data":"e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d"} Nov 01 15:58:14.379853 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.379722 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" event={"ID":"4c728783-d532-41ca-9ea3-b50340b395cd","Type":"ContainerStarted","Data":"8dbb525730365a0b17b6e1744162d2fb692ee5c55873f2c20d9501a837d53766"} Nov 01 15:58:14.400004 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.399954 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podStartSLOduration=1.399940879 podStartE2EDuration="1.399940879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:14.397690285 +0000 UTC m=+634.542144927" watchObservedRunningTime="2026-11-01 15:58:14.399940879 +0000 UTC m=+634.544395519" Nov 01 15:58:14.433301 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.433261 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx"] Nov 01 15:58:14.438723 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:14.438692 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67ab33f_6a56_48a9_a02a_5e156aed78c7.slice/crio-83d278bd244c35a5597b6fb2f6c5260e3cbab76360be69a52571b2ec1e3122bd WatchSource:0}: Error finding container 83d278bd244c35a5597b6fb2f6c5260e3cbab76360be69a52571b2ec1e3122bd: Status 404 returned error can't find the container with id 83d278bd244c35a5597b6fb2f6c5260e3cbab76360be69a52571b2ec1e3122bd Nov 01 15:58:14.441177 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.441140 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.441283 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.441254 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.441326 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.441295 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.590776 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.590542 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh"] Nov 01 15:58:14.598150 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:14.598098 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b79f2c_6850_468e_8994_0c409eead207.slice/crio-dc442a1e29c51d4c3655602d471dcadacc164bd9ef18afa3ea2a9cd9129c6213 WatchSource:0}: Error finding container dc442a1e29c51d4c3655602d471dcadacc164bd9ef18afa3ea2a9cd9129c6213: Status 404 returned error can't find the container with id dc442a1e29c51d4c3655602d471dcadacc164bd9ef18afa3ea2a9cd9129c6213 Nov 01 15:58:14.598383 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.598366 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:14.617119 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.616376 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.617119 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.616543 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.617119 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:14.616625 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:15.387165 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:15.387126 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" event={"ID":"72b79f2c-6850-468e-8994-0c409eead207","Type":"ContainerStarted","Data":"ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9"} Nov 01 15:58:15.387704 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:15.387224 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" event={"ID":"72b79f2c-6850-468e-8994-0c409eead207","Type":"ContainerStarted","Data":"dc442a1e29c51d4c3655602d471dcadacc164bd9ef18afa3ea2a9cd9129c6213"} Nov 01 15:58:15.389304 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:15.389268 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" event={"ID":"a67ab33f-6a56-48a9-a02a-5e156aed78c7","Type":"ContainerStarted","Data":"d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0"} Nov 01 15:58:15.389437 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:15.389331 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" event={"ID":"a67ab33f-6a56-48a9-a02a-5e156aed78c7","Type":"ContainerStarted","Data":"83d278bd244c35a5597b6fb2f6c5260e3cbab76360be69a52571b2ec1e3122bd"} Nov 01 15:58:15.408962 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:15.408911 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podStartSLOduration=2.408892701 podStartE2EDuration="2.408892701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:15.407881355 +0000 UTC m=+635.552335996" watchObservedRunningTime="2026-11-01 15:58:15.408892701 +0000 UTC m=+635.553347336" Nov 01 15:58:15.427106 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:15.427054 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podStartSLOduration=2.42702835 podStartE2EDuration="2.42702835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:15.425625938 +0000 UTC m=+635.570080582" watchObservedRunningTime="2026-11-01 15:58:15.42702835 +0000 UTC m=+635.571482990" Nov 01 15:58:15.618492 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:15.618456 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:16.141851 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.141813 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:16.147883 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.147853 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:16.303009 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.302973 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:16.309482 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.309453 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:16.395638 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.395550 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:16.395638 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.395601 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:16.395638 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.395625 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:16.396372 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.396351 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:16.396453 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.396403 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:16.396802 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:16.396787 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:18.978799 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:18.978761 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6"] Nov 01 15:58:18.982821 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:18.982798 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:18.993904 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:18.993866 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-164db6\"" Nov 01 15:58:18.994459 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:18.994429 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6"] Nov 01 15:58:19.007331 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.007296 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wl5\" (UniqueName: \"kubernetes.io/projected/3475b4c0-cc38-47d7-b344-8c2170eee46e-kube-api-access-t8wl5\") pod \"payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:19.007554 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.007536 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475b4c0-cc38-47d7-b344-8c2170eee46e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:19.081730 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.081696 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-164db6-dockercfg-tdx2p\"" Nov 01 15:58:19.108663 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.108627 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475b4c0-cc38-47d7-b344-8c2170eee46e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:19.108859 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.108721 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wl5\" (UniqueName: \"kubernetes.io/projected/3475b4c0-cc38-47d7-b344-8c2170eee46e-kube-api-access-t8wl5\") pod \"payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:19.109425 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.109404 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475b4c0-cc38-47d7-b344-8c2170eee46e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:19.117791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.117765 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wl5\" (UniqueName: \"kubernetes.io/projected/3475b4c0-cc38-47d7-b344-8c2170eee46e-kube-api-access-t8wl5\") pod \"payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:19.298070 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.298009 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:19.585307 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.585282 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6"] Nov 01 15:58:19.587080 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:19.587049 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3475b4c0_cc38_47d7_b344_8c2170eee46e.slice/crio-5d548131d6eafe0106431bf7927ca9c5d07019507be175fec0ea7ed1e47a0da8 WatchSource:0}: Error finding container 5d548131d6eafe0106431bf7927ca9c5d07019507be175fec0ea7ed1e47a0da8: Status 404 returned error can't find the container with id 5d548131d6eafe0106431bf7927ca9c5d07019507be175fec0ea7ed1e47a0da8 Nov 01 15:58:19.589102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:19.589080 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:58:20.419625 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.419559 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" event={"ID":"3475b4c0-cc38-47d7-b344-8c2170eee46e","Type":"ContainerStarted","Data":"5d548131d6eafe0106431bf7927ca9c5d07019507be175fec0ea7ed1e47a0da8"} Nov 01 15:58:20.738973 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.738937 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl"] Nov 01 15:58:20.743379 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.743357 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.755578 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.755543 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl"] Nov 01 15:58:20.758658 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.757934 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 15:58:20.760809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.758436 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 15:58:20.760809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.758530 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 15:58:20.760809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.758569 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-4a083d-serving-cert\"" Nov 01 15:58:20.761151 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.758638 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 15:58:20.832139 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.832106 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.832339 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.832279 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91743d76-57e1-41b8-af00-9bac5b4ce470-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.832339 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.832333 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-metrics-certs\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.832447 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.832376 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtf8\" (UniqueName: \"kubernetes.io/projected/91743d76-57e1-41b8-af00-9bac5b4ce470-kube-api-access-swtf8\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.933932 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.933889 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91743d76-57e1-41b8-af00-9bac5b4ce470-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.934135 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.933955 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-metrics-certs\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.934135 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.934002 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtf8\" (UniqueName: \"kubernetes.io/projected/91743d76-57e1-41b8-af00-9bac5b4ce470-kube-api-access-swtf8\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.934135 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.934105 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.934894 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.934847 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91743d76-57e1-41b8-af00-9bac5b4ce470-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.940429 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.940399 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-metrics-certs\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.943123 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.943096 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtf8\" (UniqueName: \"kubernetes.io/projected/91743d76-57e1-41b8-af00-9bac5b4ce470-kube-api-access-swtf8\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:20.944756 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:20.944734 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:21.075636 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.075556 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:21.368121 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.367979 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl"] Nov 01 15:58:21.427899 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.427850 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" event={"ID":"91743d76-57e1-41b8-af00-9bac5b4ce470","Type":"ContainerStarted","Data":"f586e83315b90922c923bbbf0fce2ca4f8ee4c2e9ad5afce5ce7ece77b2d6e38"} Nov 01 15:58:21.816771 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.816737 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx"] Nov 01 15:58:21.821467 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.821444 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:21.827970 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.827939 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9f4dc9-dockercfg-vrlwn\"" Nov 01 15:58:21.832659 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.832636 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx"] Nov 01 15:58:21.847953 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.847920 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:21.848115 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.848033 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4jlz\" (UniqueName: \"kubernetes.io/projected/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-kube-api-access-z4jlz\") pod \"payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:21.905722 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.905686 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9f4dc9\"" Nov 01 15:58:21.957327 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.956526 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:21.957327 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.956627 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4jlz\" (UniqueName: \"kubernetes.io/projected/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-kube-api-access-z4jlz\") pod \"payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:21.961087 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.958220 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:21.991059 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:21.977107 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4jlz\" (UniqueName: \"kubernetes.io/projected/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-kube-api-access-z4jlz\") pod \"payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:22.140189 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.139909 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:22.428371 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.428342 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx"] Nov 01 15:58:22.430480 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:22.430446 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ea7536_d3c6_4425_89c1_5ba1a36eaa7e.slice/crio-3e618b3977aac6dd2cf4f12c138d13d47bc56b6748eac92db5dcd47be5b55393 WatchSource:0}: Error finding container 3e618b3977aac6dd2cf4f12c138d13d47bc56b6748eac92db5dcd47be5b55393: Status 404 returned error can't find the container with id 3e618b3977aac6dd2cf4f12c138d13d47bc56b6748eac92db5dcd47be5b55393 Nov 01 15:58:22.437835 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.437756 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" event={"ID":"3475b4c0-cc38-47d7-b344-8c2170eee46e","Type":"ContainerStarted","Data":"188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589"} Nov 01 15:58:22.437977 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.437954 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:22.440112 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.440086 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" event={"ID":"91743d76-57e1-41b8-af00-9bac5b4ce470","Type":"ContainerStarted","Data":"51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9"} Nov 01 15:58:22.440236 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.440147 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:22.456752 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.456706 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" podStartSLOduration=1.951381295 podStartE2EDuration="4.45669105s" totalImagesPullingTime="2.505309755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:22.455781837 +0000 UTC m=+642.600236480" watchObservedRunningTime="2026-11-01 15:58:22.45669105 +0000 UTC m=+642.601145692" Nov 01 15:58:22.479963 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:22.479908 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" podStartSLOduration=2.053248725 podStartE2EDuration="2.479893488s" totalImagesPullingTime="426.644763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:22.478962156 +0000 UTC m=+642.623416827" watchObservedRunningTime="2026-11-01 15:58:22.479893488 +0000 UTC m=+642.624348129" Nov 01 15:58:23.449244 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:23.449203 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" event={"ID":"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e","Type":"ContainerStarted","Data":"e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7"} Nov 01 15:58:23.449741 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:23.449294 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" event={"ID":"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e","Type":"ContainerStarted","Data":"3e618b3977aac6dd2cf4f12c138d13d47bc56b6748eac92db5dcd47be5b55393"} Nov 01 15:58:23.449741 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:23.449329 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:23.467653 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:23.466386 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" podStartSLOduration=2.46636833 podStartE2EDuration="2.46636833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:23.464293871 +0000 UTC m=+643.608748513" watchObservedRunningTime="2026-11-01 15:58:23.46636833 +0000 UTC m=+643.610822976" Nov 01 15:58:24.020376 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.020333 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv"] Nov 01 15:58:24.028278 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.028253 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.035962 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.035927 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv"] Nov 01 15:58:24.036250 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.036227 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-bd831a-serving-cert\"" Nov 01 15:58:24.080777 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.080746 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2c301e-c797-47bf-81e6-16251407a79e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.080919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.080782 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bh5b\" (UniqueName: \"kubernetes.io/projected/6f2c301e-c797-47bf-81e6-16251407a79e-kube-api-access-8bh5b\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.080919 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.080868 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.081021 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.080948 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-metrics-certs\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.181548 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.181510 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-metrics-certs\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.181726 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.181602 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2c301e-c797-47bf-81e6-16251407a79e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.181726 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.181622 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bh5b\" (UniqueName: \"kubernetes.io/projected/6f2c301e-c797-47bf-81e6-16251407a79e-kube-api-access-8bh5b\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.181726 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.181660 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.181830 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:58:24.181754 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-serving-cert: secret "maas-api-e2e-worker-w3-bd831a-serving-cert" not found Nov 01 15:58:24.181830 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:58:24.181813 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls podName:6f2c301e-c797-47bf-81e6-16251407a79e nodeName:}" failed. No retries permitted until 2026-11-01 15:58:24.681798132 +0000 UTC m=+644.826252754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls") pod "maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" (UID: "6f2c301e-c797-47bf-81e6-16251407a79e") : secret "maas-api-e2e-worker-w3-bd831a-serving-cert" not found Nov 01 15:58:24.182421 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.182393 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2c301e-c797-47bf-81e6-16251407a79e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.184078 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.184056 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-metrics-certs\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.190278 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.190250 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bh5b\" (UniqueName: \"kubernetes.io/projected/6f2c301e-c797-47bf-81e6-16251407a79e-kube-api-access-8bh5b\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.686464 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.686424 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.688917 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.688885 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:24.949119 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:24.948968 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:25.229942 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.229909 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv"] Nov 01 15:58:25.231906 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:25.231869 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2c301e_c797_47bf_81e6_16251407a79e.slice/crio-ab6114c380068986b76a96eb751e958614e325e67f1db456365b86942a86d979 WatchSource:0}: Error finding container ab6114c380068986b76a96eb751e958614e325e67f1db456365b86942a86d979: Status 404 returned error can't find the container with id ab6114c380068986b76a96eb751e958614e325e67f1db456365b86942a86d979 Nov 01 15:58:25.464114 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.464072 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" event={"ID":"6f2c301e-c797-47bf-81e6-16251407a79e","Type":"ContainerStarted","Data":"ab6114c380068986b76a96eb751e958614e325e67f1db456365b86942a86d979"} Nov 01 15:58:25.628696 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.628664 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg"] Nov 01 15:58:25.633702 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.633677 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.642709 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.642669 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg"] Nov 01 15:58:25.645649 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.644791 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-dfff59-serving-cert\"" Nov 01 15:58:25.695832 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.695801 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-metrics-certs\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.696411 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.695858 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.696411 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.695971 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cead9c-dd49-4156-9505-0c0d4d195843-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.696411 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.696029 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr7vm\" (UniqueName: \"kubernetes.io/projected/56cead9c-dd49-4156-9505-0c0d4d195843-kube-api-access-sr7vm\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.796557 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.796520 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-metrics-certs\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.796743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.796584 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.796743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.796631 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cead9c-dd49-4156-9505-0c0d4d195843-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.796743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.796686 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr7vm\" (UniqueName: \"kubernetes.io/projected/56cead9c-dd49-4156-9505-0c0d4d195843-kube-api-access-sr7vm\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.797310 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.797283 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cead9c-dd49-4156-9505-0c0d4d195843-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.799005 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.798982 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.799125 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.799076 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-metrics-certs\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.804558 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.804537 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr7vm\" (UniqueName: \"kubernetes.io/projected/56cead9c-dd49-4156-9505-0c0d4d195843-kube-api-access-sr7vm\") pod \"maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:25.953458 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:25.953370 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:26.233105 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.232992 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg"] Nov 01 15:58:26.239028 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:26.238997 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cead9c_dd49_4156_9505_0c0d4d195843.slice/crio-faafdca2120f26ca770f352e676eba938ec38b349c03dbfd35a2bc3a5f2fa336 WatchSource:0}: Error finding container faafdca2120f26ca770f352e676eba938ec38b349c03dbfd35a2bc3a5f2fa336: Status 404 returned error can't find the container with id faafdca2120f26ca770f352e676eba938ec38b349c03dbfd35a2bc3a5f2fa336 Nov 01 15:58:26.473900 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.473862 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" event={"ID":"6f2c301e-c797-47bf-81e6-16251407a79e","Type":"ContainerStarted","Data":"f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549"} Nov 01 15:58:26.474099 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.473950 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:26.475818 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.475783 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" event={"ID":"56cead9c-dd49-4156-9505-0c0d4d195843","Type":"ContainerStarted","Data":"faafdca2120f26ca770f352e676eba938ec38b349c03dbfd35a2bc3a5f2fa336"} Nov 01 15:58:26.493665 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.493618 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" podStartSLOduration=3.055563491 podStartE2EDuration="3.493603374s" totalImagesPullingTime="438.039883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:26.491704823 +0000 UTC m=+646.636159487" watchObservedRunningTime="2026-11-01 15:58:26.493603374 +0000 UTC m=+646.638058014" Nov 01 15:58:26.632549 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.632512 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl"] Nov 01 15:58:26.639066 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.639005 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.657282 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.657239 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-9f4dc9-serving-cert\"" Nov 01 15:58:26.661140 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.661102 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl"] Nov 01 15:58:26.704553 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.704514 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.705002 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.704661 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.705002 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.704803 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-metrics-certs\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.705002 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.704922 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmchs\" (UniqueName: \"kubernetes.io/projected/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-kube-api-access-hmchs\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.808898 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.808800 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-metrics-certs\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.809743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.809713 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmchs\" (UniqueName: \"kubernetes.io/projected/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-kube-api-access-hmchs\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.809933 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.809917 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.810107 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.810089 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.810394 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:58:26.810378 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-serving-cert: secret "maas-api-e2e-worker-w4-9f4dc9-serving-cert" not found Nov 01 15:58:26.810601 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:58:26.810591 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls podName:1c7dbfa6-cea0-4fd7-ae29-471160afbd7d nodeName:}" failed. No retries permitted until 2026-11-01 15:58:27.310568209 +0000 UTC m=+647.455022844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls") pod "maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" (UID: "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d") : secret "maas-api-e2e-worker-w4-9f4dc9-serving-cert" not found Nov 01 15:58:26.811878 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.811841 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.821237 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.821178 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmchs\" (UniqueName: \"kubernetes.io/projected/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-kube-api-access-hmchs\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:26.825939 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:26.825911 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-metrics-certs\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:27.316167 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:27.316125 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:27.318567 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:27.318531 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:27.490309 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:27.490273 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" event={"ID":"56cead9c-dd49-4156-9505-0c0d4d195843","Type":"ContainerStarted","Data":"5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911"} Nov 01 15:58:27.490491 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:27.490373 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:27.514815 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:27.514768 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" podStartSLOduration=1.899002428 podStartE2EDuration="2.514751979s" totalImagesPullingTime="615.749551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:27.512117882 +0000 UTC m=+647.656572522" watchObservedRunningTime="2026-11-01 15:58:27.514751979 +0000 UTC m=+647.659206620" Nov 01 15:58:27.565064 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:27.563601 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:28.037394 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:28.037356 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7dbfa6_cea0_4fd7_ae29_471160afbd7d.slice/crio-87cea3b99d06f54061863f72a02d2251fe5eda903211bee40b2dcf21ebcc13e2 WatchSource:0}: Error finding container 87cea3b99d06f54061863f72a02d2251fe5eda903211bee40b2dcf21ebcc13e2: Status 404 returned error can't find the container with id 87cea3b99d06f54061863f72a02d2251fe5eda903211bee40b2dcf21ebcc13e2 Nov 01 15:58:28.039379 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:28.039217 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl"] Nov 01 15:58:28.485133 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:28.484222 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 15:58:28.548600 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:28.548566 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" event={"ID":"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d","Type":"ContainerStarted","Data":"87cea3b99d06f54061863f72a02d2251fe5eda903211bee40b2dcf21ebcc13e2"} Nov 01 15:58:29.579211 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:29.578644 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" event={"ID":"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d","Type":"ContainerStarted","Data":"e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07"} Nov 01 15:58:29.579211 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:29.579110 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:29.600061 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:29.598306 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" podStartSLOduration=2.992678472 podStartE2EDuration="3.598287478s" totalImagesPullingTime="605.609006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:29.596035327 +0000 UTC m=+649.740489973" watchObservedRunningTime="2026-11-01 15:58:29.598287478 +0000 UTC m=+649.742742120" Nov 01 15:58:32.936066 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:32.928683 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 15:58:33.491772 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:33.491742 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 15:58:33.598685 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:33.593828 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 15:58:34.459127 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:34.459091 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 15:58:35.812900 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:35.812860 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 15:58:39.506572 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:39.035834 4911 generic.go:356] "Generic (PLEG): container finished" podID="4c728783-d532-41ca-9ea3-b50340b395cd" containerID="e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d" exitCode=0 Nov 01 15:58:39.506572 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:39.035912 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" event={"ID":"4c728783-d532-41ca-9ea3-b50340b395cd","Type":"ContainerDied","Data":"e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d"} Nov 01 15:58:39.506572 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:39.036347 4911 scope.go:122] "RemoveContainer" containerID="e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d" Nov 01 15:58:39.506572 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:39.037726 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:39.506572 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:39.037785 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:39.506572 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:39.037818 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.044102 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.044060 4911 generic.go:356] "Generic (PLEG): container finished" podID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerID="d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0" exitCode=0 Nov 01 15:58:40.044337 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.044129 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" event={"ID":"a67ab33f-6a56-48a9-a02a-5e156aed78c7","Type":"ContainerDied","Data":"d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0"} Nov 01 15:58:40.044602 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.044581 4911 scope.go:122] "RemoveContainer" containerID="d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0" Nov 01 15:58:40.045928 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.045894 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.046054 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.045972 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.046054 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.046007 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.046598 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.046579 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" event={"ID":"4c728783-d532-41ca-9ea3-b50340b395cd","Type":"ContainerStarted","Data":"6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce"} Nov 01 15:58:40.048346 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.048326 4911 generic.go:356] "Generic (PLEG): container finished" podID="72b79f2c-6850-468e-8994-0c409eead207" containerID="ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9" exitCode=0 Nov 01 15:58:40.048449 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.048393 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" event={"ID":"72b79f2c-6850-468e-8994-0c409eead207","Type":"ContainerDied","Data":"ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9"} Nov 01 15:58:40.048812 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.048794 4911 scope.go:122] "RemoveContainer" containerID="ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9" Nov 01 15:58:40.063060 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.062740 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.063060 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.062837 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.063060 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.062878 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:40.600063 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.599462 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:40.600597 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.600266 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:58:40.600597 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:40.600325 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 01 15:58:40.934145 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:58:40.934025 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:58:41.103070 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:41.093426 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" event={"ID":"72b79f2c-6850-468e-8994-0c409eead207","Type":"ContainerStarted","Data":"f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987"} Nov 01 15:58:41.145313 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:41.145278 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" event={"ID":"a67ab33f-6a56-48a9-a02a-5e156aed78c7","Type":"ContainerStarted","Data":"a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc"} Nov 01 15:58:41.306063 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:41.305107 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:41.306063 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:41.305268 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 15:58:41.306063 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:41.305319 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 15:58:41.598123 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:41.596476 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:58:41.598123 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:41.596541 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 01 15:58:42.141077 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:42.141017 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:42.142311 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:42.142283 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:58:42.142489 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:42.142469 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 01 15:58:42.303453 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:42.303406 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 15:58:42.303738 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:42.303711 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 15:58:42.601063 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:42.596463 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:58:42.601063 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:42.596528 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 01 15:58:43.141837 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:43.141798 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:58:43.142251 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:43.141855 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 01 15:58:43.303472 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:43.303431 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 15:58:43.303766 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:43.303733 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 15:58:43.601479 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:43.601439 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:58:43.601791 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:43.601756 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 01 15:58:44.144921 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.144882 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:58:44.145560 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.145511 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 01 15:58:44.306518 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.306475 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 15:58:44.306806 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.306765 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 15:58:44.596601 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.596564 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:58:44.596900 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.596860 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 01 15:58:44.799086 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.703148 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8"] Nov 01 15:58:44.799086 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.751509 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.824076 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.823596 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"kube-root-ca.crt\"" Nov 01 15:58:44.824076 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.823781 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"default-dockercfg-l8c2c\"" Nov 01 15:58:44.824076 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.823882 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"simulator-w5-164db6-kserve-self-signed-certs\"" Nov 01 15:58:44.824076 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.823954 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"openshift-service-ca.crt\"" Nov 01 15:58:44.886503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.873854 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8"] Nov 01 15:58:44.886503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.878772 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-422qm\" (UniqueName: \"kubernetes.io/projected/474d2b5a-53c8-43f8-8892-23f3d843febb-kube-api-access-422qm\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.886503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.878851 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-dshm\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.886503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.878949 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-home\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.886503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.879009 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-model-cache\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.886503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.879063 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-tmp-dir\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.886503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.879104 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d2b5a-53c8-43f8-8892-23f3d843febb-tls-certs\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.981080 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.981020 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-dshm\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.981417 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.981395 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-home\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.981580 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.981562 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-model-cache\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.981712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.981696 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-tmp-dir\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.981836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.981821 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d2b5a-53c8-43f8-8892-23f3d843febb-tls-certs\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.981969 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.981953 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-422qm\" (UniqueName: \"kubernetes.io/projected/474d2b5a-53c8-43f8-8892-23f3d843febb-kube-api-access-422qm\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.983241 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.983213 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-model-cache\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.984896 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.983648 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-tmp-dir\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:44.985069 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.984825 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-home\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:45.003070 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:44.987827 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d2b5a-53c8-43f8-8892-23f3d843febb-tls-certs\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:45.003619 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.003589 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-dshm\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:45.016707 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.016675 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-422qm\" (UniqueName: \"kubernetes.io/projected/474d2b5a-53c8-43f8-8892-23f3d843febb-kube-api-access-422qm\") pod \"simulator-w5-164db6-kserve-696555ff9c-wnsg8\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:45.141367 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.141284 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:58:45.141596 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.141571 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 01 15:58:45.190257 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.190214 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:45.308684 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.308234 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 15:58:45.308684 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.308297 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 15:58:45.539614 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.539576 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8"] Nov 01 15:58:45.541178 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:45.541095 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod474d2b5a_53c8_43f8_8892_23f3d843febb.slice/crio-47b77d27900b0c2fc28cd5b70c54d303bb78c37cc9c69dbe2edf900bac64e1ea WatchSource:0}: Error finding container 47b77d27900b0c2fc28cd5b70c54d303bb78c37cc9c69dbe2edf900bac64e1ea: Status 404 returned error can't find the container with id 47b77d27900b0c2fc28cd5b70c54d303bb78c37cc9c69dbe2edf900bac64e1ea Nov 01 15:58:45.621933 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.621264 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:45.622320 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.622247 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:45.622880 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:45.622857 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 15:58:46.142080 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:46.142021 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:58:46.142278 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:46.142106 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 01 15:58:46.227260 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:46.227216 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" event={"ID":"474d2b5a-53c8-43f8-8892-23f3d843febb","Type":"ContainerStarted","Data":"47b77d27900b0c2fc28cd5b70c54d303bb78c37cc9c69dbe2edf900bac64e1ea"} Nov 01 15:58:46.303503 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:46.303462 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 01 15:58:46.303712 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:46.303525 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 01 15:58:46.401557 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:46.400268 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:46.401557 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:46.401032 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:47.150079 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:47.147710 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:47.236507 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:47.236477 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 15:58:47.307825 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:47.307797 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:47.308922 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:47.308898 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 15:58:48.242549 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:48.242509 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" event={"ID":"474d2b5a-53c8-43f8-8892-23f3d843febb","Type":"ContainerStarted","Data":"cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c"} Nov 01 15:58:48.259081 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:48.259005 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podStartSLOduration=2.445410386 podStartE2EDuration="4.25899126s" totalImagesPullingTime="1.813580874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:48.258315934 +0000 UTC m=+668.402770575" watchObservedRunningTime="2026-11-01 15:58:48.25899126 +0000 UTC m=+668.403445900" Nov 01 15:58:49.191374 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:49.191335 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:51.349959 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.349868 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh"] Nov 01 15:58:51.355034 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.354999 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.362011 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.361984 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"kube-root-ca.crt\"" Nov 01 15:58:51.362397 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.362369 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh"] Nov 01 15:58:51.362840 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.362725 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"default-dockercfg-z2zvx\"" Nov 01 15:58:51.364164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.363153 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"openshift-service-ca.crt\"" Nov 01 15:58:51.364841 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.364359 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"simulator-w6-43fac8-kserve-self-signed-certs\"" Nov 01 15:58:51.401762 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.401728 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-model-cache\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.401966 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.401861 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-dshm\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.402150 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.401977 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-home\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.402150 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.402092 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-tmp-dir\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.402150 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.402141 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc428f9-1f81-49ae-8aee-5f82900fc489-tls-certs\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.402337 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.402166 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldlkp\" (UniqueName: \"kubernetes.io/projected/1cc428f9-1f81-49ae-8aee-5f82900fc489-kube-api-access-ldlkp\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.503658 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.503624 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc428f9-1f81-49ae-8aee-5f82900fc489-tls-certs\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.503658 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.503665 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldlkp\" (UniqueName: \"kubernetes.io/projected/1cc428f9-1f81-49ae-8aee-5f82900fc489-kube-api-access-ldlkp\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.503910 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.503715 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-model-cache\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.503910 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.503788 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-dshm\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.503910 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.503823 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-home\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.504116 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.503961 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-tmp-dir\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.504264 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.504240 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-model-cache\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.504333 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.504302 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-home\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.504467 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.504436 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-tmp-dir\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.506401 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.506374 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-dshm\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.506680 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.506660 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc428f9-1f81-49ae-8aee-5f82900fc489-tls-certs\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.511838 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.511812 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldlkp\" (UniqueName: \"kubernetes.io/projected/1cc428f9-1f81-49ae-8aee-5f82900fc489-kube-api-access-ldlkp\") pod \"simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.672374 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.672291 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:51.948622 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:51.948595 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh"] Nov 01 15:58:51.950347 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:58:51.950303 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cc428f9_1f81_49ae_8aee_5f82900fc489.slice/crio-70790a0e4fb8125881eedf73acdf7a6c9dd0047614b52f97536be1a146fdeb32 WatchSource:0}: Error finding container 70790a0e4fb8125881eedf73acdf7a6c9dd0047614b52f97536be1a146fdeb32: Status 404 returned error can't find the container with id 70790a0e4fb8125881eedf73acdf7a6c9dd0047614b52f97536be1a146fdeb32 Nov 01 15:58:52.272558 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.272524 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" event={"ID":"1cc428f9-1f81-49ae-8aee-5f82900fc489","Type":"ContainerStarted","Data":"603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905"} Nov 01 15:58:52.272775 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.272574 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" event={"ID":"1cc428f9-1f81-49ae-8aee-5f82900fc489","Type":"ContainerStarted","Data":"70790a0e4fb8125881eedf73acdf7a6c9dd0047614b52f97536be1a146fdeb32"} Nov 01 15:58:52.290601 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.290554 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podStartSLOduration=1.290540614 podStartE2EDuration="1.290540614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:52.28898901 +0000 UTC m=+672.433443652" watchObservedRunningTime="2026-11-01 15:58:52.290540614 +0000 UTC m=+672.434995255" Nov 01 15:58:52.387732 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.387696 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h"] Nov 01 15:58:52.391787 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.391759 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.401639 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.401607 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h"] Nov 01 15:58:52.401944 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.401922 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"openshift-service-ca.crt\"" Nov 01 15:58:52.402095 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.401983 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"kube-root-ca.crt\"" Nov 01 15:58:52.402095 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.402072 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"simulator-w1-dfff59-kserve-self-signed-certs\"" Nov 01 15:58:52.402197 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.402148 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"default-dockercfg-4dxd5\"" Nov 01 15:58:52.519147 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.519116 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-model-cache\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.519317 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.519159 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-tmp-dir\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.519317 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.519186 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9153161a-1192-4086-b3c5-8dcced55123f-tls-certs\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.519410 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.519324 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-home\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.519476 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.519450 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzds\" (UniqueName: \"kubernetes.io/projected/9153161a-1192-4086-b3c5-8dcced55123f-kube-api-access-nmzds\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.519622 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.519603 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-dshm\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.620862 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.620770 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-dshm\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.620862 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.620811 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-model-cache\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.620862 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.620835 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-tmp-dir\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.620862 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.620858 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9153161a-1192-4086-b3c5-8dcced55123f-tls-certs\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.621232 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.620928 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-home\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.621232 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.621010 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzds\" (UniqueName: \"kubernetes.io/projected/9153161a-1192-4086-b3c5-8dcced55123f-kube-api-access-nmzds\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.621437 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.621413 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-model-cache\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.621504 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.621446 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-home\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.621589 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.621571 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-tmp-dir\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.623153 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.623123 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-dshm\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.623416 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.623399 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9153161a-1192-4086-b3c5-8dcced55123f-tls-certs\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.628788 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.628765 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzds\" (UniqueName: \"kubernetes.io/projected/9153161a-1192-4086-b3c5-8dcced55123f-kube-api-access-nmzds\") pod \"simulator-w1-dfff59-kserve-8cf5d89c6-wx48h\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.673222 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.673178 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:58:52.707476 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.707442 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:52.990911 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:52.990871 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h"] Nov 01 15:58:53.279905 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:53.279800 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" event={"ID":"9153161a-1192-4086-b3c5-8dcced55123f","Type":"ContainerStarted","Data":"cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c"} Nov 01 15:58:53.279905 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:53.279850 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" event={"ID":"9153161a-1192-4086-b3c5-8dcced55123f","Type":"ContainerStarted","Data":"e23d33b19647e35f984c00c202baa24a0de88a9ab8f3b20e1812e4eceedc2541"} Nov 01 15:58:53.298704 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:53.298653 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podStartSLOduration=1.298641063 podStartE2EDuration="1.298641063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:53.296008975 +0000 UTC m=+673.440463620" watchObservedRunningTime="2026-11-01 15:58:53.298641063 +0000 UTC m=+673.443095703" Nov 01 15:58:53.707724 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:53.707635 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:58:55.191571 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:55.191533 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:55.205176 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:55.205147 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:58:55.310203 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:58:55.310172 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 15:59:01.691072 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:01.691017 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:59:01.721364 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:01.721332 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:59:02.355836 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:02.355801 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 15:59:02.708494 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:02.708412 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:59:02.721273 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:02.721241 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:59:03.359350 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:03.359322 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 15:59:10.866339 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:59:10.866303 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:59:13.255656 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.255608 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl"] Nov 01 15:59:13.262842 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.262815 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl"] Nov 01 15:59:13.263232 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.263159 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.276516 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.276484 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"premium-w2-6b0e0f-kserve-self-signed-certs\"" Nov 01 15:59:13.276981 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.276820 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"openshift-service-ca.crt\"" Nov 01 15:59:13.277271 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.277252 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"kube-root-ca.crt\"" Nov 01 15:59:13.278216 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.278180 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"default-dockercfg-2f26v\"" Nov 01 15:59:13.357387 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.357353 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-tmp-dir\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.357605 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.357429 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-home\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.357605 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.357465 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-model-cache\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.357605 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.357490 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-dshm\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.357605 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.357537 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e441066e-613b-4f47-be65-0e60deb73fc2-tls-certs\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.357815 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.357649 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks65v\" (UniqueName: \"kubernetes.io/projected/e441066e-613b-4f47-be65-0e60deb73fc2-kube-api-access-ks65v\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459150 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459034 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks65v\" (UniqueName: \"kubernetes.io/projected/e441066e-613b-4f47-be65-0e60deb73fc2-kube-api-access-ks65v\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459329 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459168 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-tmp-dir\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459329 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459226 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-home\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459329 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459258 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-model-cache\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459329 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459284 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-dshm\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459651 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459334 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e441066e-613b-4f47-be65-0e60deb73fc2-tls-certs\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459829 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459800 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-model-cache\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.459902 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.459830 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-tmp-dir\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.460164 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.460140 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-home\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.462745 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.462716 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e441066e-613b-4f47-be65-0e60deb73fc2-tls-certs\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.462935 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.462915 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-dshm\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.468484 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.468458 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks65v\" (UniqueName: \"kubernetes.io/projected/e441066e-613b-4f47-be65-0e60deb73fc2-kube-api-access-ks65v\") pod \"premium-w2-6b0e0f-kserve-779cc8d955-mlcjl\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.595778 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.595671 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:13.921900 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:13.921866 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl"] Nov 01 15:59:14.425434 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:14.425399 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" event={"ID":"e441066e-613b-4f47-be65-0e60deb73fc2","Type":"ContainerStarted","Data":"380bcd066cb147c98ca38e4a86fcaa1c93cf6579d36ea78d9f87c34e679246f3"} Nov 01 15:59:14.425944 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:14.425443 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" event={"ID":"e441066e-613b-4f47-be65-0e60deb73fc2","Type":"ContainerStarted","Data":"3f6b6857f7c43764af81f80fc401fc79cc6e9774588488872cc113ad019989da"} Nov 01 15:59:14.444345 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:14.444297 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podStartSLOduration=1.4442836780000001 podStartE2EDuration="1.444283678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:14.441784278 +0000 UTC m=+694.586238919" watchObservedRunningTime="2026-11-01 15:59:14.444283678 +0000 UTC m=+694.588738319" Nov 01 15:59:14.596324 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:14.596282 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:16.830952 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:16.830922 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6"] Nov 01 15:59:16.835328 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:16.835302 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:16.842178 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:16.842146 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6"] Nov 01 15:59:16.844839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:16.844819 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"premium-w6-43fac8-kserve-self-signed-certs\"" Nov 01 15:59:17.002823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.002783 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x8c\" (UniqueName: \"kubernetes.io/projected/783e58a7-525d-43b7-8da4-0be7c41efd4f-kube-api-access-42x8c\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.003001 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.002830 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-dshm\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.003001 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.002941 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-home\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.003001 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.002984 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-model-cache\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.003183 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.003116 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-tmp-dir\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.003286 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.003269 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783e58a7-525d-43b7-8da4-0be7c41efd4f-tls-certs\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.104731 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.104645 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783e58a7-525d-43b7-8da4-0be7c41efd4f-tls-certs\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.104915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.104752 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x8c\" (UniqueName: \"kubernetes.io/projected/783e58a7-525d-43b7-8da4-0be7c41efd4f-kube-api-access-42x8c\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.104915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.104780 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-dshm\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.104915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.104840 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-home\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.105134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.105030 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-model-cache\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.105134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.105115 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-tmp-dir\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.105446 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.105417 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-home\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.105581 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.105451 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-model-cache\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.105655 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.105622 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-tmp-dir\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.107204 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.107179 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-dshm\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.107539 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.107520 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783e58a7-525d-43b7-8da4-0be7c41efd4f-tls-certs\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.113195 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.113169 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x8c\" (UniqueName: \"kubernetes.io/projected/783e58a7-525d-43b7-8da4-0be7c41efd4f-kube-api-access-42x8c\") pod \"premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.166929 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.166889 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:17.447035 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.446953 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6"] Nov 01 15:59:17.451211 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:59:17.451178 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod783e58a7_525d_43b7_8da4_0be7c41efd4f.slice/crio-ed748f83071d766adfddf2eec761f149434f030c4e5e8cfced407833e667ed9e WatchSource:0}: Error finding container ed748f83071d766adfddf2eec761f149434f030c4e5e8cfced407833e667ed9e: Status 404 returned error can't find the container with id ed748f83071d766adfddf2eec761f149434f030c4e5e8cfced407833e667ed9e Nov 01 15:59:17.618705 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.618659 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc"] Nov 01 15:59:17.625121 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.623511 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.634291 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.627733 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc"] Nov 01 15:59:17.637424 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.637394 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"premium-w1-dfff59-kserve-self-signed-certs\"" Nov 01 15:59:17.716928 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.716827 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-home\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.716928 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.716885 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170897de-1a22-4f8f-be3b-d4b1b4322663-tls-certs\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.717202 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.716985 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-model-cache\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.717202 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.717033 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgqf\" (UniqueName: \"kubernetes.io/projected/170897de-1a22-4f8f-be3b-d4b1b4322663-kube-api-access-wbgqf\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.717313 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.717209 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-tmp-dir\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.717313 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.717278 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-dshm\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.818903 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.818854 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170897de-1a22-4f8f-be3b-d4b1b4322663-tls-certs\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.818914 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-model-cache\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.818950 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgqf\" (UniqueName: \"kubernetes.io/projected/170897de-1a22-4f8f-be3b-d4b1b4322663-kube-api-access-wbgqf\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819143 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.819126 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-tmp-dir\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819318 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.819178 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-dshm\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819318 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.819251 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-home\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819425 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.819400 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-model-cache\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819496 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.819457 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-tmp-dir\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.819686 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.819661 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-home\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.821453 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.821423 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-dshm\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.821584 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.821566 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170897de-1a22-4f8f-be3b-d4b1b4322663-tls-certs\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.828661 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.828632 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgqf\" (UniqueName: \"kubernetes.io/projected/170897de-1a22-4f8f-be3b-d4b1b4322663-kube-api-access-wbgqf\") pod \"premium-w1-dfff59-kserve-6c46b6d789-gs5jc\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:17.950232 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:17.950197 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:18.229704 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.229673 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc"] Nov 01 15:59:18.230266 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:59:18.230231 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod170897de_1a22_4f8f_be3b_d4b1b4322663.slice/crio-1638b85f6ea7789f9a47fe66d50ff4f4cd9ebe38c74a1e1db7bf3d3930c81497 WatchSource:0}: Error finding container 1638b85f6ea7789f9a47fe66d50ff4f4cd9ebe38c74a1e1db7bf3d3930c81497: Status 404 returned error can't find the container with id 1638b85f6ea7789f9a47fe66d50ff4f4cd9ebe38c74a1e1db7bf3d3930c81497 Nov 01 15:59:18.451790 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.451754 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" event={"ID":"170897de-1a22-4f8f-be3b-d4b1b4322663","Type":"ContainerStarted","Data":"cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529"} Nov 01 15:59:18.451993 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.451807 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" event={"ID":"170897de-1a22-4f8f-be3b-d4b1b4322663","Type":"ContainerStarted","Data":"1638b85f6ea7789f9a47fe66d50ff4f4cd9ebe38c74a1e1db7bf3d3930c81497"} Nov 01 15:59:18.453964 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.453934 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" event={"ID":"783e58a7-525d-43b7-8da4-0be7c41efd4f","Type":"ContainerStarted","Data":"33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3"} Nov 01 15:59:18.454110 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.453971 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" event={"ID":"783e58a7-525d-43b7-8da4-0be7c41efd4f","Type":"ContainerStarted","Data":"ed748f83071d766adfddf2eec761f149434f030c4e5e8cfced407833e667ed9e"} Nov 01 15:59:18.471337 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.471290 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podStartSLOduration=1.471276652 podStartE2EDuration="1.471276652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:18.46963125 +0000 UTC m=+698.614085927" watchObservedRunningTime="2026-11-01 15:59:18.471276652 +0000 UTC m=+698.615731638" Nov 01 15:59:18.496075 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.489855 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podStartSLOduration=2.489840213 podStartE2EDuration="2.489840213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:18.486297789 +0000 UTC m=+698.630752434" watchObservedRunningTime="2026-11-01 15:59:18.489840213 +0000 UTC m=+698.634294855" Nov 01 15:59:18.950643 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:18.950607 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:19.167605 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:19.167571 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:23.595972 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:23.595929 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:23.612054 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:23.612009 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:24.512688 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:24.512656 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 15:59:27.167823 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:27.167793 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:27.180743 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:27.180715 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:27.531594 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:27.531560 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 15:59:27.951379 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:27.951290 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:27.965273 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:27.965243 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:28.539627 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:28.539600 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 15:59:40.865777 ip-10-0-143-101 kubenswrapper[4911]: E1101 15:59:40.865742 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 15:59:43.855095 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:43.854997 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp"] Nov 01 15:59:43.860505 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:43.860481 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:43.868800 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:43.868573 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9f4dc9-openshift-default-dockercfg-tmp6r\"" Nov 01 15:59:43.873089 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:43.873062 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp"] Nov 01 15:59:44.055604 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.055564 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.055604 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.055608 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.055871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.055626 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.055871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.055795 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.055871 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.055837 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.056027 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.055947 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.056027 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.056005 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6k4m\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-kube-api-access-m6k4m\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.056156 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.056082 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.056190 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.056147 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.157670 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.157568 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.157670 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.157630 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6k4m\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-kube-api-access-m6k4m\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.157915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.157703 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.157915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.157785 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.157915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.157860 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.157915 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.157894 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158157 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.157921 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158157 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.158007 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158157 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.158035 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158520 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.158497 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158717 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.158650 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.158650 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.158650 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.158839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.158747 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.161230 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.161204 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.161421 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.161398 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.166397 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.166371 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.166982 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.166956 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6k4m\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-kube-api-access-m6k4m\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.176809 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.176784 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:44.457839 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.457809 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp"] Nov 01 15:59:44.458368 ip-10-0-143-101 kubenswrapper[4911]: W1101 15:59:44.458339 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d55996_1e97_4797_ba6a_31a5ff5974c4.slice/crio-e6dee911609c6c3b12768ea817a32673d7be6f98da14cbb67d4a52f022924668 WatchSource:0}: Error finding container e6dee911609c6c3b12768ea817a32673d7be6f98da14cbb67d4a52f022924668: Status 404 returned error can't find the container with id e6dee911609c6c3b12768ea817a32673d7be6f98da14cbb67d4a52f022924668 Nov 01 15:59:44.460819 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.460789 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:44.460880 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.460854 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:44.460923 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.460885 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:44.657544 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.657500 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" event={"ID":"b8d55996-1e97-4797-ba6a-31a5ff5974c4","Type":"ContainerStarted","Data":"5adb8de432cb3461427a34d2b42bb6901674140479af7fca8349c6b93dadd9e1"} Nov 01 15:59:44.657714 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.657566 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" event={"ID":"b8d55996-1e97-4797-ba6a-31a5ff5974c4","Type":"ContainerStarted","Data":"e6dee911609c6c3b12768ea817a32673d7be6f98da14cbb67d4a52f022924668"} Nov 01 15:59:44.676319 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:44.676264 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podStartSLOduration=1.676251218 podStartE2EDuration="1.676251218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:44.675355779 +0000 UTC m=+724.819810446" watchObservedRunningTime="2026-11-01 15:59:44.676251218 +0000 UTC m=+724.820705933" Nov 01 15:59:45.177219 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:45.177184 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:45.179219 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:45.179181 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 15:59:45.179389 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:45.179240 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 15:59:46.177707 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:46.177660 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 15:59:46.178134 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:46.177752 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 15:59:47.179064 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:47.177389 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 15:59:47.179064 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:47.177454 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 15:59:48.184891 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:48.184854 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 15:59:48.185582 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:48.185550 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 15:59:49.178064 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:49.177293 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 15:59:49.178064 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:49.177363 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 15:59:50.179507 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:50.179464 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 15:59:50.180194 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:50.180152 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 15:59:51.178033 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:51.177978 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 01 15:59:51.178274 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:51.178099 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 01 15:59:52.487470 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:52.487435 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:52.487975 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:52.487855 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 15:59:52.489985 ip-10-0-143-101 kubenswrapper[4911]: I1101 15:59:52.489963 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 16:00:00.169794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.169758 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7"] Nov 01 16:00:00.174026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.174001 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.183545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.183506 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7"] Nov 01 16:00:00.215986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.215955 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.216186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.216000 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrzm\" (UniqueName: \"kubernetes.io/projected/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-kube-api-access-ldrzm\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.254094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.254061 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:00:00.317082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.317024 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.317278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.317113 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrzm\" (UniqueName: \"kubernetes.io/projected/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-kube-api-access-ldrzm\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.317729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.317708 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.324767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.324742 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrzm\" (UniqueName: \"kubernetes.io/projected/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-kube-api-access-ldrzm\") pod \"maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.494677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.494645 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:00.774131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.774094 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7"] Nov 01 16:00:00.775458 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:00.775432 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd25c13d_2335_4d32_9ca0_e3bf0c47979d.slice/crio-61a2f1745d0a1c3fffae950683a994c2a1279d9df931d4feb2a94fc9ef21281d WatchSource:0}: Error finding container 61a2f1745d0a1c3fffae950683a994c2a1279d9df931d4feb2a94fc9ef21281d: Status 404 returned error can't find the container with id 61a2f1745d0a1c3fffae950683a994c2a1279d9df931d4feb2a94fc9ef21281d Nov 01 16:00:00.846292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:00.846253 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" event={"ID":"cd25c13d-2335-4d32-9ca0-e3bf0c47979d","Type":"ContainerStarted","Data":"61a2f1745d0a1c3fffae950683a994c2a1279d9df931d4feb2a94fc9ef21281d"} Nov 01 16:00:05.908462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:05.908410 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" event={"ID":"cd25c13d-2335-4d32-9ca0-e3bf0c47979d","Type":"ContainerStarted","Data":"2bc589c984e515a330ab6a764f389feff4e6e30ea3b671026dbd32ca2389a63a"} Nov 01 16:00:05.932829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:05.932775 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" podStartSLOduration=1.873429524 podStartE2EDuration="5.932760795s" totalImagesPullingTime="4.059331271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:05.929656893 +0000 UTC m=+746.074111533" watchObservedRunningTime="2026-11-01 16:00:05.932760795 +0000 UTC m=+746.077215475" Nov 01 16:00:05.997490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:05.997451 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb"] Nov 01 16:00:06.001534 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.001506 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.006383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.006358 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:06.007747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.007720 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb"] Nov 01 16:00:06.088140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.088097 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwnlt\" (UniqueName: \"kubernetes.io/projected/84e31a76-a70f-4d9c-825a-9e141962cbca-kube-api-access-bwnlt\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.088307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.088156 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.088307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.088263 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.189027 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.188946 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwnlt\" (UniqueName: \"kubernetes.io/projected/84e31a76-a70f-4d9c-825a-9e141962cbca-kube-api-access-bwnlt\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.189027 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.189020 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.189248 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.189102 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.189889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.189860 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.191699 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.191673 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.196983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.196962 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwnlt\" (UniqueName: \"kubernetes.io/projected/84e31a76-a70f-4d9c-825a-9e141962cbca-kube-api-access-bwnlt\") pod \"maas-api-revoke-sub-c218dc07-jfjnb\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.339395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.339361 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:06.641574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.641540 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb"] Nov 01 16:00:06.642094 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:06.642031 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e31a76_a70f_4d9c_825a_9e141962cbca.slice/crio-caab32239ec594b8b06fe9e483e95fe1ecac610e6d2a666a20be969e28796a05 WatchSource:0}: Error finding container caab32239ec594b8b06fe9e483e95fe1ecac610e6d2a666a20be969e28796a05: Status 404 returned error can't find the container with id caab32239ec594b8b06fe9e483e95fe1ecac610e6d2a666a20be969e28796a05 Nov 01 16:00:06.918448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.918359 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" event={"ID":"84e31a76-a70f-4d9c-825a-9e141962cbca","Type":"ContainerStarted","Data":"caab32239ec594b8b06fe9e483e95fe1ecac610e6d2a666a20be969e28796a05"} Nov 01 16:00:06.920125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.920094 4911 generic.go:356] "Generic (PLEG): container finished" podID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerID="2bc589c984e515a330ab6a764f389feff4e6e30ea3b671026dbd32ca2389a63a" exitCode=60 Nov 01 16:00:06.920269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.920130 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" event={"ID":"cd25c13d-2335-4d32-9ca0-e3bf0c47979d","Type":"ContainerDied","Data":"2bc589c984e515a330ab6a764f389feff4e6e30ea3b671026dbd32ca2389a63a"} Nov 01 16:00:06.920445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:06.920430 4911 scope.go:122] "RemoveContainer" containerID="2bc589c984e515a330ab6a764f389feff4e6e30ea3b671026dbd32ca2389a63a" Nov 01 16:00:07.935124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:07.931965 4911 generic.go:356] "Generic (PLEG): container finished" podID="84e31a76-a70f-4d9c-825a-9e141962cbca" containerID="61976262b7dbac595ad5ee342c45431b9403c92b6cd1ebc2a83f69b79957d6c6" exitCode=0 Nov 01 16:00:07.935124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:07.932091 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" event={"ID":"84e31a76-a70f-4d9c-825a-9e141962cbca","Type":"ContainerDied","Data":"61976262b7dbac595ad5ee342c45431b9403c92b6cd1ebc2a83f69b79957d6c6"} Nov 01 16:00:07.935669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:07.935520 4911 generic.go:356] "Generic (PLEG): container finished" podID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerID="9d332b55f2dc30e84e69e04e19504ede645b9558f0b7a7ecc23c31d62693f515" exitCode=60 Nov 01 16:00:07.935669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:07.935582 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" event={"ID":"cd25c13d-2335-4d32-9ca0-e3bf0c47979d","Type":"ContainerDied","Data":"9d332b55f2dc30e84e69e04e19504ede645b9558f0b7a7ecc23c31d62693f515"} Nov 01 16:00:07.935669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:07.935628 4911 scope.go:122] "RemoveContainer" containerID="2bc589c984e515a330ab6a764f389feff4e6e30ea3b671026dbd32ca2389a63a" Nov 01 16:00:07.936124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:07.936105 4911 scope.go:122] "RemoveContainer" containerID="9d332b55f2dc30e84e69e04e19504ede645b9558f0b7a7ecc23c31d62693f515" Nov 01 16:00:07.936431 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:00:07.936369 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7_odh-ai-gateway-infra(cd25c13d-2335-4d32-9ca0-e3bf0c47979d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" Nov 01 16:00:08.948448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:08.947289 4911 scope.go:122] "RemoveContainer" containerID="9d332b55f2dc30e84e69e04e19504ede645b9558f0b7a7ecc23c31d62693f515" Nov 01 16:00:08.948448 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:00:08.947644 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7_odh-ai-gateway-infra(cd25c13d-2335-4d32-9ca0-e3bf0c47979d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" Nov 01 16:00:09.119347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.119316 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:09.269389 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.269339 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-xks-ca\") pod \"84e31a76-a70f-4d9c-825a-9e141962cbca\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " Nov 01 16:00:09.269389 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.269393 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e31a76-a70f-4d9c-825a-9e141962cbca-kube-api-access-bwnlt\" (UniqueName: \"kubernetes.io/projected/84e31a76-a70f-4d9c-825a-9e141962cbca-kube-api-access-bwnlt\") pod \"84e31a76-a70f-4d9c-825a-9e141962cbca\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " Nov 01 16:00:09.269611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.269425 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-openshift-ca\") pod \"84e31a76-a70f-4d9c-825a-9e141962cbca\" (UID: \"84e31a76-a70f-4d9c-825a-9e141962cbca\") " Nov 01 16:00:09.269914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.269886 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-openshift-ca" pod "84e31a76-a70f-4d9c-825a-9e141962cbca" (UID: "84e31a76-a70f-4d9c-825a-9e141962cbca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:09.270024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.269988 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:09.271930 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.271903 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-xks-ca" pod "84e31a76-a70f-4d9c-825a-9e141962cbca" (UID: "84e31a76-a70f-4d9c-825a-9e141962cbca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:09.272201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.272177 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e31a76-a70f-4d9c-825a-9e141962cbca-kube-api-access-bwnlt" pod "84e31a76-a70f-4d9c-825a-9e141962cbca" (UID: "84e31a76-a70f-4d9c-825a-9e141962cbca"). InnerVolumeSpecName "kube-api-access-bwnlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:09.370842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.370805 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e31a76-a70f-4d9c-825a-9e141962cbca-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:09.370842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.370840 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwnlt\" (UniqueName: \"kubernetes.io/projected/84e31a76-a70f-4d9c-825a-9e141962cbca-kube-api-access-bwnlt\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:09.955239 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.955202 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" Nov 01 16:00:09.955655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.955235 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb" event={"ID":"84e31a76-a70f-4d9c-825a-9e141962cbca","Type":"ContainerDied","Data":"caab32239ec594b8b06fe9e483e95fe1ecac610e6d2a666a20be969e28796a05"} Nov 01 16:00:09.955655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:09.955316 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caab32239ec594b8b06fe9e483e95fe1ecac610e6d2a666a20be969e28796a05" Nov 01 16:00:10.918744 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:00:10.916754 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:00:11.985913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:11.985876 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5"] Nov 01 16:00:11.986440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:11.986424 4911 cpu_manager.go:414] "containerMap: removing container" podUID="84e31a76-a70f-4d9c-825a-9e141962cbca" containerName="revoke-keys" Nov 01 16:00:11.986485 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:11.986440 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e31a76-a70f-4d9c-825a-9e141962cbca" containerName="revoke-keys" Nov 01 16:00:11.986574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:11.986564 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e31a76-a70f-4d9c-825a-9e141962cbca" containerName="revoke-keys" Nov 01 16:00:11.989977 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:11.989962 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:11.993548 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:11.993353 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:11.996301 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:11.996280 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5"] Nov 01 16:00:12.024857 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.024825 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.025016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.024940 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxfq\" (UniqueName: \"kubernetes.io/projected/48fe9127-45e5-4751-b211-65f91a376efe-kube-api-access-wkxfq\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.025169 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.025144 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.126592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.126555 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.126783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.126642 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.126783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.126714 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxfq\" (UniqueName: \"kubernetes.io/projected/48fe9127-45e5-4751-b211-65f91a376efe-kube-api-access-wkxfq\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.127305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.127279 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.128952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.128931 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.137193 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.137170 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxfq\" (UniqueName: \"kubernetes.io/projected/48fe9127-45e5-4751-b211-65f91a376efe-kube-api-access-wkxfq\") pod \"maas-api-revoke-sub-d29cbb63-p4wz5\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.309898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.309811 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:12.587325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.587299 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5"] Nov 01 16:00:12.589518 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:12.589482 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fe9127_45e5_4751_b211_65f91a376efe.slice/crio-bc0000d76858f080818033249117b8dbdf07c2d79df7c33d0887318e393ce38f WatchSource:0}: Error finding container bc0000d76858f080818033249117b8dbdf07c2d79df7c33d0887318e393ce38f: Status 404 returned error can't find the container with id bc0000d76858f080818033249117b8dbdf07c2d79df7c33d0887318e393ce38f Nov 01 16:00:12.981730 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.981691 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" event={"ID":"48fe9127-45e5-4751-b211-65f91a376efe","Type":"ContainerStarted","Data":"98f2007a8a2773dc05044e83fdbdcd4ec1bd217b2db7c7bb7966985bf958409a"} Nov 01 16:00:12.981906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:12.981750 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" event={"ID":"48fe9127-45e5-4751-b211-65f91a376efe","Type":"ContainerStarted","Data":"bc0000d76858f080818033249117b8dbdf07c2d79df7c33d0887318e393ce38f"} Nov 01 16:00:13.001332 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:13.001279 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" podStartSLOduration=2.001265351 podStartE2EDuration="2.001265351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:12.998078466 +0000 UTC m=+753.142533106" watchObservedRunningTime="2026-11-01 16:00:13.001265351 +0000 UTC m=+753.145719991" Nov 01 16:00:13.991245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:13.991207 4911 generic.go:356] "Generic (PLEG): container finished" podID="48fe9127-45e5-4751-b211-65f91a376efe" containerID="98f2007a8a2773dc05044e83fdbdcd4ec1bd217b2db7c7bb7966985bf958409a" exitCode=0 Nov 01 16:00:13.991798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:13.991299 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" event={"ID":"48fe9127-45e5-4751-b211-65f91a376efe","Type":"ContainerDied","Data":"98f2007a8a2773dc05044e83fdbdcd4ec1bd217b2db7c7bb7966985bf958409a"} Nov 01 16:00:15.167264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.167236 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:15.269015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.268927 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fe9127-45e5-4751-b211-65f91a376efe-kube-api-access-wkxfq\" (UniqueName: \"kubernetes.io/projected/48fe9127-45e5-4751-b211-65f91a376efe-kube-api-access-wkxfq\") pod \"48fe9127-45e5-4751-b211-65f91a376efe\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " Nov 01 16:00:15.269181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.269033 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-openshift-ca\") pod \"48fe9127-45e5-4751-b211-65f91a376efe\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " Nov 01 16:00:15.269181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.269163 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-xks-ca\") pod \"48fe9127-45e5-4751-b211-65f91a376efe\" (UID: \"48fe9127-45e5-4751-b211-65f91a376efe\") " Nov 01 16:00:15.269398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.269372 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-openshift-ca" pod "48fe9127-45e5-4751-b211-65f91a376efe" (UID: "48fe9127-45e5-4751-b211-65f91a376efe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:15.269609 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.269594 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:15.271085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.271060 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fe9127-45e5-4751-b211-65f91a376efe-kube-api-access-wkxfq" pod "48fe9127-45e5-4751-b211-65f91a376efe" (UID: "48fe9127-45e5-4751-b211-65f91a376efe"). InnerVolumeSpecName "kube-api-access-wkxfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:15.271175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.271128 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-xks-ca" pod "48fe9127-45e5-4751-b211-65f91a376efe" (UID: "48fe9127-45e5-4751-b211-65f91a376efe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:15.371278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.371242 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fe9127-45e5-4751-b211-65f91a376efe-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:15.371278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:15.371274 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxfq\" (UniqueName: \"kubernetes.io/projected/48fe9127-45e5-4751-b211-65f91a376efe-kube-api-access-wkxfq\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:16.006170 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:16.006137 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" Nov 01 16:00:16.006311 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:16.006162 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5" event={"ID":"48fe9127-45e5-4751-b211-65f91a376efe","Type":"ContainerDied","Data":"bc0000d76858f080818033249117b8dbdf07c2d79df7c33d0887318e393ce38f"} Nov 01 16:00:16.006311 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:16.006212 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0000d76858f080818033249117b8dbdf07c2d79df7c33d0887318e393ce38f" Nov 01 16:00:17.034123 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.034086 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq"] Nov 01 16:00:17.034579 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.034566 4911 cpu_manager.go:414] "containerMap: removing container" podUID="48fe9127-45e5-4751-b211-65f91a376efe" containerName="revoke-keys" Nov 01 16:00:17.034628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.034580 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fe9127-45e5-4751-b211-65f91a376efe" containerName="revoke-keys" Nov 01 16:00:17.034692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.034683 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fe9127-45e5-4751-b211-65f91a376efe" containerName="revoke-keys" Nov 01 16:00:17.038209 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.038191 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.044708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.044681 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:17.048167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.048142 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq"] Nov 01 16:00:17.091256 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.091218 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthpg\" (UniqueName: \"kubernetes.io/projected/792d8743-937e-4de1-b29e-48b04c55e69c-kube-api-access-wthpg\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.091418 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.091394 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.091615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.091595 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.192720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.192687 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.192889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.192730 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthpg\" (UniqueName: \"kubernetes.io/projected/792d8743-937e-4de1-b29e-48b04c55e69c-kube-api-access-wthpg\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.192889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.192801 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.193456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.193425 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.195212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.195191 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.202312 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.202286 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthpg\" (UniqueName: \"kubernetes.io/projected/792d8743-937e-4de1-b29e-48b04c55e69c-kube-api-access-wthpg\") pod \"maas-api-revoke-sub-a8f35e65-b4bcq\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.358712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.358626 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:17.643606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:17.643575 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq"] Nov 01 16:00:17.644694 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:17.644662 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792d8743_937e_4de1_b29e_48b04c55e69c.slice/crio-864617b837754c33170f7033c2feba673fc825a3e8ad1d81348d7dcffbc763ae WatchSource:0}: Error finding container 864617b837754c33170f7033c2feba673fc825a3e8ad1d81348d7dcffbc763ae: Status 404 returned error can't find the container with id 864617b837754c33170f7033c2feba673fc825a3e8ad1d81348d7dcffbc763ae Nov 01 16:00:18.020836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:18.020800 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" event={"ID":"792d8743-937e-4de1-b29e-48b04c55e69c","Type":"ContainerStarted","Data":"f8ff99335288cf94265f01d5641c1e13d911fc531e19e8cfdef4a474b106edf4"} Nov 01 16:00:18.020979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:18.020849 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" event={"ID":"792d8743-937e-4de1-b29e-48b04c55e69c","Type":"ContainerStarted","Data":"864617b837754c33170f7033c2feba673fc825a3e8ad1d81348d7dcffbc763ae"} Nov 01 16:00:18.038400 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:18.038347 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" podStartSLOduration=1.038333573 podStartE2EDuration="1.038333573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:18.035964399 +0000 UTC m=+758.180419037" watchObservedRunningTime="2026-11-01 16:00:18.038333573 +0000 UTC m=+758.182788248" Nov 01 16:00:19.028593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:19.028549 4911 generic.go:356] "Generic (PLEG): container finished" podID="792d8743-937e-4de1-b29e-48b04c55e69c" containerID="f8ff99335288cf94265f01d5641c1e13d911fc531e19e8cfdef4a474b106edf4" exitCode=0 Nov 01 16:00:19.028768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:19.028609 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" event={"ID":"792d8743-937e-4de1-b29e-48b04c55e69c","Type":"ContainerDied","Data":"f8ff99335288cf94265f01d5641c1e13d911fc531e19e8cfdef4a474b106edf4"} Nov 01 16:00:19.965116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:19.965085 4911 scope.go:122] "RemoveContainer" containerID="9d332b55f2dc30e84e69e04e19504ede645b9558f0b7a7ecc23c31d62693f515" Nov 01 16:00:20.218639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.218572 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:20.328692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.328658 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792d8743-937e-4de1-b29e-48b04c55e69c-kube-api-access-wthpg\" (UniqueName: \"kubernetes.io/projected/792d8743-937e-4de1-b29e-48b04c55e69c-kube-api-access-wthpg\") pod \"792d8743-937e-4de1-b29e-48b04c55e69c\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " Nov 01 16:00:20.328881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.328755 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-openshift-ca\") pod \"792d8743-937e-4de1-b29e-48b04c55e69c\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " Nov 01 16:00:20.328881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.328831 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-xks-ca\") pod \"792d8743-937e-4de1-b29e-48b04c55e69c\" (UID: \"792d8743-937e-4de1-b29e-48b04c55e69c\") " Nov 01 16:00:20.329174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.329144 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-openshift-ca" pod "792d8743-937e-4de1-b29e-48b04c55e69c" (UID: "792d8743-937e-4de1-b29e-48b04c55e69c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:20.329272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.329242 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:20.330881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.330857 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792d8743-937e-4de1-b29e-48b04c55e69c-kube-api-access-wthpg" pod "792d8743-937e-4de1-b29e-48b04c55e69c" (UID: "792d8743-937e-4de1-b29e-48b04c55e69c"). InnerVolumeSpecName "kube-api-access-wthpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:20.330982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.330902 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-xks-ca" pod "792d8743-937e-4de1-b29e-48b04c55e69c" (UID: "792d8743-937e-4de1-b29e-48b04c55e69c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:20.429870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.429833 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wthpg\" (UniqueName: \"kubernetes.io/projected/792d8743-937e-4de1-b29e-48b04c55e69c-kube-api-access-wthpg\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:20.429870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:20.429866 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d8743-937e-4de1-b29e-48b04c55e69c-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:21.043844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:21.043755 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" Nov 01 16:00:21.043844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:21.043792 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq" event={"ID":"792d8743-937e-4de1-b29e-48b04c55e69c","Type":"ContainerDied","Data":"864617b837754c33170f7033c2feba673fc825a3e8ad1d81348d7dcffbc763ae"} Nov 01 16:00:21.044349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:21.043851 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="864617b837754c33170f7033c2feba673fc825a3e8ad1d81348d7dcffbc763ae" Nov 01 16:00:21.045764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:21.045742 4911 generic.go:356] "Generic (PLEG): container finished" podID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerID="d5881bf62e38a0813931257a4cba86f77bba3f49f2f1298713029ab8f1097bcc" exitCode=60 Nov 01 16:00:21.045894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:21.045815 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" event={"ID":"cd25c13d-2335-4d32-9ca0-e3bf0c47979d","Type":"ContainerDied","Data":"d5881bf62e38a0813931257a4cba86f77bba3f49f2f1298713029ab8f1097bcc"} Nov 01 16:00:21.045894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:21.045874 4911 scope.go:122] "RemoveContainer" containerID="9d332b55f2dc30e84e69e04e19504ede645b9558f0b7a7ecc23c31d62693f515" Nov 01 16:00:21.046202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:21.046179 4911 scope.go:122] "RemoveContainer" containerID="d5881bf62e38a0813931257a4cba86f77bba3f49f2f1298713029ab8f1097bcc" Nov 01 16:00:21.046441 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:00:21.046424 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7_odh-ai-gateway-infra(cd25c13d-2335-4d32-9ca0-e3bf0c47979d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" Nov 01 16:00:22.076871 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.076839 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7"] Nov 01 16:00:22.125696 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.125648 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk"] Nov 01 16:00:22.126634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.126605 4911 cpu_manager.go:414] "containerMap: removing container" podUID="792d8743-937e-4de1-b29e-48b04c55e69c" containerName="revoke-keys" Nov 01 16:00:22.126634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.126632 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792d8743-937e-4de1-b29e-48b04c55e69c" containerName="revoke-keys" Nov 01 16:00:22.126829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.126811 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="792d8743-937e-4de1-b29e-48b04c55e69c" containerName="revoke-keys" Nov 01 16:00:22.131888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.131862 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.136815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.136790 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk"] Nov 01 16:00:22.140965 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.140942 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:22.257889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.257851 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcb7\" (UniqueName: \"kubernetes.io/projected/ac1e9534-870d-431f-b22d-935f969c3e45-kube-api-access-5lcb7\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.257889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.257893 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.258146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.258083 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.271482 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.271456 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:22.359201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.359128 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-kube-api-access-ldrzm\" (UniqueName: \"kubernetes.io/projected/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-kube-api-access-ldrzm\") pod \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " Nov 01 16:00:22.359348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.359309 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-maas-api-service-ca\") pod \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\" (UID: \"cd25c13d-2335-4d32-9ca0-e3bf0c47979d\") " Nov 01 16:00:22.359620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.359595 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.359778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.359658 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-maas-api-service-ca" pod "cd25c13d-2335-4d32-9ca0-e3bf0c47979d" (UID: "cd25c13d-2335-4d32-9ca0-e3bf0c47979d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:22.359843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.359802 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcb7\" (UniqueName: \"kubernetes.io/projected/ac1e9534-870d-431f-b22d-935f969c3e45-kube-api-access-5lcb7\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.359897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.359848 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.360024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.360009 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:22.360637 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.360610 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.361431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.361407 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-kube-api-access-ldrzm" pod "cd25c13d-2335-4d32-9ca0-e3bf0c47979d" (UID: "cd25c13d-2335-4d32-9ca0-e3bf0c47979d"). InnerVolumeSpecName "kube-api-access-ldrzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:22.362201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.362171 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.378076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.376650 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcb7\" (UniqueName: \"kubernetes.io/projected/ac1e9534-870d-431f-b22d-935f969c3e45-kube-api-access-5lcb7\") pod \"maas-api-revoke-sub-eeac48cb-bnwkk\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.459231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.459196 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:22.460678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.460647 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrzm\" (UniqueName: \"kubernetes.io/projected/cd25c13d-2335-4d32-9ca0-e3bf0c47979d-kube-api-access-ldrzm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:22.739674 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:22.739641 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk"] Nov 01 16:00:22.739991 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:22.739968 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1e9534_870d_431f_b22d_935f969c3e45.slice/crio-b61634e902a4b0aeafc93e21292cdaa03a1300cdbb9835404e0a06d280256dd5 WatchSource:0}: Error finding container b61634e902a4b0aeafc93e21292cdaa03a1300cdbb9835404e0a06d280256dd5: Status 404 returned error can't find the container with id b61634e902a4b0aeafc93e21292cdaa03a1300cdbb9835404e0a06d280256dd5 Nov 01 16:00:23.063820 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:23.063798 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" Nov 01 16:00:23.063956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:23.063830 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7" event={"ID":"cd25c13d-2335-4d32-9ca0-e3bf0c47979d","Type":"ContainerDied","Data":"61a2f1745d0a1c3fffae950683a994c2a1279d9df931d4feb2a94fc9ef21281d"} Nov 01 16:00:23.063956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:23.063904 4911 scope.go:122] "RemoveContainer" containerID="d5881bf62e38a0813931257a4cba86f77bba3f49f2f1298713029ab8f1097bcc" Nov 01 16:00:23.066537 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:23.065764 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" event={"ID":"ac1e9534-870d-431f-b22d-935f969c3e45","Type":"ContainerStarted","Data":"b61634e902a4b0aeafc93e21292cdaa03a1300cdbb9835404e0a06d280256dd5"} Nov 01 16:00:23.092268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:23.092229 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7"] Nov 01 16:00:23.097186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:23.097147 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-43fac8-29892480-msjq7"] Nov 01 16:00:23.967500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:23.967426 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" path="/var/lib/kubelet/pods/cd25c13d-2335-4d32-9ca0-e3bf0c47979d/volumes" Nov 01 16:00:24.076572 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:24.076544 4911 generic.go:356] "Generic (PLEG): container finished" podID="ac1e9534-870d-431f-b22d-935f969c3e45" containerID="4964358a36fcc9bdb3aeb37b3a87446e8472b17f0791cb93656e72553ead2d18" exitCode=0 Nov 01 16:00:24.076739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:24.076612 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" event={"ID":"ac1e9534-870d-431f-b22d-935f969c3e45","Type":"ContainerDied","Data":"4964358a36fcc9bdb3aeb37b3a87446e8472b17f0791cb93656e72553ead2d18"} Nov 01 16:00:25.700063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.696492 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:25.839119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.816452 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-openshift-ca\") pod \"ac1e9534-870d-431f-b22d-935f969c3e45\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " Nov 01 16:00:25.839119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.816561 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-xks-ca\") pod \"ac1e9534-870d-431f-b22d-935f969c3e45\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " Nov 01 16:00:25.839119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.816607 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1e9534-870d-431f-b22d-935f969c3e45-kube-api-access-5lcb7\" (UniqueName: \"kubernetes.io/projected/ac1e9534-870d-431f-b22d-935f969c3e45-kube-api-access-5lcb7\") pod \"ac1e9534-870d-431f-b22d-935f969c3e45\" (UID: \"ac1e9534-870d-431f-b22d-935f969c3e45\") " Nov 01 16:00:25.839119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.820416 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1e9534-870d-431f-b22d-935f969c3e45-kube-api-access-5lcb7" pod "ac1e9534-870d-431f-b22d-935f969c3e45" (UID: "ac1e9534-870d-431f-b22d-935f969c3e45"). InnerVolumeSpecName "kube-api-access-5lcb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:25.839119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.826442 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-openshift-ca" pod "ac1e9534-870d-431f-b22d-935f969c3e45" (UID: "ac1e9534-870d-431f-b22d-935f969c3e45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:25.866064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.863562 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-xks-ca" pod "ac1e9534-870d-431f-b22d-935f969c3e45" (UID: "ac1e9534-870d-431f-b22d-935f969c3e45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:25.922065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.921892 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:25.922065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.921940 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1e9534-870d-431f-b22d-935f969c3e45-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:25.922065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:25.921960 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lcb7\" (UniqueName: \"kubernetes.io/projected/ac1e9534-870d-431f-b22d-935f969c3e45-kube-api-access-5lcb7\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.156120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:26.155866 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" event={"ID":"ac1e9534-870d-431f-b22d-935f969c3e45","Type":"ContainerDied","Data":"b61634e902a4b0aeafc93e21292cdaa03a1300cdbb9835404e0a06d280256dd5"} Nov 01 16:00:26.156120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:26.155936 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b61634e902a4b0aeafc93e21292cdaa03a1300cdbb9835404e0a06d280256dd5" Nov 01 16:00:26.156120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:26.156006 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk" Nov 01 16:00:26.207063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:26.202101 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 16:00:28.606009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.605974 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq"] Nov 01 16:00:28.606548 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606533 4911 cpu_manager.go:414] "containerMap: removing container" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.606599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606550 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.606599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606562 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ac1e9534-870d-431f-b22d-935f969c3e45" containerName="revoke-keys" Nov 01 16:00:28.606599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606569 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1e9534-870d-431f-b22d-935f969c3e45" containerName="revoke-keys" Nov 01 16:00:28.606599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606585 4911 cpu_manager.go:414] "containerMap: removing container" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.606599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606592 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.606751 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606663 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1e9534-870d-431f-b22d-935f969c3e45" containerName="revoke-keys" Nov 01 16:00:28.606751 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606673 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.606751 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606678 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.606751 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.606689 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.613734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.613713 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.618246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.618183 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4a083d-openshift-default-dockercfg-qxpcr\"" Nov 01 16:00:28.624187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.624161 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq"] Nov 01 16:00:28.724594 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.724549 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.724757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.724694 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af24b8-930b-48af-a4d9-0ee884f97bb3-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.724810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.724756 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.724810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.724787 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.724943 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.724923 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.724986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.724956 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.724986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.724982 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.725111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.725078 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.725111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.725102 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dml7r\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-kube-api-access-dml7r\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.765352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.765320 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q"] Nov 01 16:00:28.765860 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.765848 4911 cpu_manager.go:414] "containerMap: removing container" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.765904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.765862 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd25c13d-2335-4d32-9ca0-e3bf0c47979d" containerName="cleanup" Nov 01 16:00:28.770203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.770186 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.778270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.778224 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfff59-openshift-default-dockercfg-bzb6m\"" Nov 01 16:00:28.782017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.781994 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q"] Nov 01 16:00:28.825930 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.825902 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.825938 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.825969 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.825993 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dml7r\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-kube-api-access-dml7r\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826018 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826101 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af24b8-930b-48af-a4d9-0ee884f97bb3-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826139 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826169 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826237 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826341 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826541 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826397 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826579 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826547 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826602 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.826855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.826830 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af24b8-930b-48af-a4d9-0ee884f97bb3-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.828251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.828235 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.828690 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.828666 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.834393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.834367 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dml7r\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-kube-api-access-dml7r\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.834565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.834546 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:28.927977 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.927886 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.927977 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.927945 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.928244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.928102 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.928244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.928135 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4qv\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-kube-api-access-dg4qv\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.928361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.928238 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.928421 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.928407 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.928458 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.928444 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.928522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.928508 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.928566 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.928545 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:28.931858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:28.931632 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:29.030460 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030421 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030478 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030545 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030571 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4qv\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-kube-api-access-dg4qv\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030621 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030762 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030850 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030795 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030850 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030833 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.030913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.030878 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.031187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.031166 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.031311 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.031289 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.032175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.032149 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.032284 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.032261 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.032363 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.032342 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.033500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.033402 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.034586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.034538 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.039951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.039925 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.040301 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.040284 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4qv\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-kube-api-access-dg4qv\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.085150 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.085116 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:29.226266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.226235 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq"] Nov 01 16:00:29.226544 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:29.226521 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60af24b8_930b_48af_a4d9_0ee884f97bb3.slice/crio-98f49047cfda77da8aecce8b136f7a5d16f2c5f2dda9ecd969ca8d595d26f2c6 WatchSource:0}: Error finding container 98f49047cfda77da8aecce8b136f7a5d16f2c5f2dda9ecd969ca8d595d26f2c6: Status 404 returned error can't find the container with id 98f49047cfda77da8aecce8b136f7a5d16f2c5f2dda9ecd969ca8d595d26f2c6 Nov 01 16:00:29.228999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.228963 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.229112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.229072 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.229185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.229126 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.367565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.367530 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q"] Nov 01 16:00:29.367778 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:29.367753 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53180dc_7fc1_417d_93e2_23dfc9ea3673.slice/crio-ffabe99859cb6a8dc12a33b3ae8bba05218eaa5ea11a77b27468f4f161b6aca1 WatchSource:0}: Error finding container ffabe99859cb6a8dc12a33b3ae8bba05218eaa5ea11a77b27468f4f161b6aca1: Status 404 returned error can't find the container with id ffabe99859cb6a8dc12a33b3ae8bba05218eaa5ea11a77b27468f4f161b6aca1 Nov 01 16:00:29.370392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.370356 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.370482 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.370438 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.370522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:29.370478 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:30.214626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.214577 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" event={"ID":"f53180dc-7fc1-417d-93e2-23dfc9ea3673","Type":"ContainerStarted","Data":"eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097"} Nov 01 16:00:30.215153 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.214646 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" event={"ID":"f53180dc-7fc1-417d-93e2-23dfc9ea3673","Type":"ContainerStarted","Data":"ffabe99859cb6a8dc12a33b3ae8bba05218eaa5ea11a77b27468f4f161b6aca1"} Nov 01 16:00:30.216289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.216264 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" event={"ID":"60af24b8-930b-48af-a4d9-0ee884f97bb3","Type":"ContainerStarted","Data":"1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91"} Nov 01 16:00:30.216430 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.216305 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" event={"ID":"60af24b8-930b-48af-a4d9-0ee884f97bb3","Type":"ContainerStarted","Data":"98f49047cfda77da8aecce8b136f7a5d16f2c5f2dda9ecd969ca8d595d26f2c6"} Nov 01 16:00:30.237791 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.237743 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podStartSLOduration=2.237730484 podStartE2EDuration="2.237730484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:30.233682562 +0000 UTC m=+770.378137204" watchObservedRunningTime="2026-11-01 16:00:30.237730484 +0000 UTC m=+770.382185125" Nov 01 16:00:30.931859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.931818 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:30.933132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.933106 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:00:30.933267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:30.933195 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 16:00:31.096066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:31.094529 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:31.096066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:31.094660 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:00:31.096066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:31.094715 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 16:00:31.932102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:31.932071 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:00:31.932574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:31.932549 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 16:00:32.088062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:32.087444 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:00:32.088062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:32.087508 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 16:00:32.932017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:32.931981 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:00:32.932186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:32.932050 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 16:00:33.095509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:33.095471 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:00:33.095803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:33.095770 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 16:00:33.932369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:33.932334 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:00:33.932843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:33.932387 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 16:00:34.023828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.023771 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podStartSLOduration=6.023735788 podStartE2EDuration="6.023735788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:30.274158082 +0000 UTC m=+770.418612725" watchObservedRunningTime="2026-11-01 16:00:34.023735788 +0000 UTC m=+774.168190431" Nov 01 16:00:34.025066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.024831 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5"] Nov 01 16:00:34.030174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.030149 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5"] Nov 01 16:00:34.030316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.030251 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.034508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.034483 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"kube-root-ca.crt\"" Nov 01 16:00:34.040026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.035498 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"openshift-service-ca.crt\"" Nov 01 16:00:34.040026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.035633 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"default-dockercfg-n49qh\"" Nov 01 16:00:34.040026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.035780 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"distinct-w4-9f4dc9-kserve-self-signed-certs\"" Nov 01 16:00:34.104191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.104149 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:00:34.104465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.104429 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 16:00:34.183307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.183185 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-model-cache\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.183671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.183624 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-dshm\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.183898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.183877 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tmp-dir\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.184156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.184101 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr82f\" (UniqueName: \"kubernetes.io/projected/2ca8e5b5-b21b-4719-966a-e48aac6919ad-kube-api-access-mr82f\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.184395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.184376 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tls-certs\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.185442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.185422 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-home\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.287869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.287816 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tls-certs\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.288236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.288202 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-home\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.288516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.288484 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-model-cache\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.288661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.288647 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-dshm\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.288791 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.288775 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tmp-dir\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.288956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.288924 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr82f\" (UniqueName: \"kubernetes.io/projected/2ca8e5b5-b21b-4719-966a-e48aac6919ad-kube-api-access-mr82f\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.290506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.290484 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-home\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.291284 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.290905 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tmp-dir\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.291736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.291715 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-model-cache\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.300440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.300409 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-dshm\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.314191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.309874 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr82f\" (UniqueName: \"kubernetes.io/projected/2ca8e5b5-b21b-4719-966a-e48aac6919ad-kube-api-access-mr82f\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.330933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.330903 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tls-certs\") pod \"distinct-w4-9f4dc9-kserve-577bc6974-ztqc5\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.374313 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.374269 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:34.829085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.815523 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5"] Nov 01 16:00:34.934102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.933814 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:00:34.934102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:34.933882 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 16:00:35.088547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.088452 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 01 16:00:35.088801 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.088771 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 01 16:00:35.307032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.306989 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" event={"ID":"2ca8e5b5-b21b-4719-966a-e48aac6919ad","Type":"ContainerStarted","Data":"d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639"} Nov 01 16:00:35.307300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.307276 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" event={"ID":"2ca8e5b5-b21b-4719-966a-e48aac6919ad","Type":"ContainerStarted","Data":"8132a8361798019b686588f9c0b046093a4e7f6af61fb23b70892b94ca3d5aa0"} Nov 01 16:00:35.332560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.332502 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podStartSLOduration=2.332488041 podStartE2EDuration="2.332488041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:35.329958971 +0000 UTC m=+775.474413607" watchObservedRunningTime="2026-11-01 16:00:35.332488041 +0000 UTC m=+775.476942681" Nov 01 16:00:35.389421 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.378139 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:35.932519 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.932476 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 01 16:00:35.932675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:35.932551 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 01 16:00:36.339368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:36.339331 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:36.340287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:36.340259 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:36.341551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:36.341530 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:00:36.938726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:36.938693 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:36.939009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:36.938984 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:36.939726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:36.939706 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:00:40.942062 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:00:40.932727 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:00:44.375522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.375484 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:44.389414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.389386 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:44.429501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.429387 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs"] Nov 01 16:00:44.435249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.435226 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.449564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.442580 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs"] Nov 01 16:00:44.449564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.442899 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"distinct-w2-6b0e0f-kserve-self-signed-certs\"" Nov 01 16:00:44.483827 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.483775 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-home\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.483982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.483849 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-model-cache\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.484081 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.483972 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tls-certs\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.484139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.484078 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tmp-dir\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.484178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.484157 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmbr\" (UniqueName: \"kubernetes.io/projected/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-kube-api-access-xjmbr\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.484507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.484479 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-dshm\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.585797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.585749 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-home\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.585983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.585824 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-model-cache\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.585983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.585870 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tls-certs\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.585983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.585910 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tmp-dir\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.585983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.585965 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmbr\" (UniqueName: \"kubernetes.io/projected/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-kube-api-access-xjmbr\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.586289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.586093 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-dshm\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.587944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.587915 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tmp-dir\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.588374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.588349 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-home\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.588749 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.588729 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-model-cache\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.620445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.620403 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmbr\" (UniqueName: \"kubernetes.io/projected/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-kube-api-access-xjmbr\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.621272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.621245 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tls-certs\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.621503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.621482 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-dshm\") pod \"distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:44.754966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:44.754929 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:45.052881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:45.052850 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs"] Nov 01 16:00:45.054153 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:45.054124 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod686a5cc9_26a2_44e2_8a1a_ba5001f492b7.slice/crio-213293bacfb827ccd6b5934a25ba24de1df604ff6d2316ed122ab8afe34c9aaf WatchSource:0}: Error finding container 213293bacfb827ccd6b5934a25ba24de1df604ff6d2316ed122ab8afe34c9aaf: Status 404 returned error can't find the container with id 213293bacfb827ccd6b5934a25ba24de1df604ff6d2316ed122ab8afe34c9aaf Nov 01 16:00:45.390238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:45.390154 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" event={"ID":"686a5cc9-26a2-44e2-8a1a-ba5001f492b7","Type":"ContainerStarted","Data":"bfae6e904d4f5b826f34f3258993dc346f115b98628346a5aaa3acea6582b6fb"} Nov 01 16:00:45.390238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:45.390202 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" event={"ID":"686a5cc9-26a2-44e2-8a1a-ba5001f492b7","Type":"ContainerStarted","Data":"213293bacfb827ccd6b5934a25ba24de1df604ff6d2316ed122ab8afe34c9aaf"} Nov 01 16:00:45.393105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:45.393077 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:00:45.412519 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:45.412473 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podStartSLOduration=1.412457531 podStartE2EDuration="1.412457531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:45.408088658 +0000 UTC m=+785.552543301" watchObservedRunningTime="2026-11-01 16:00:45.412457531 +0000 UTC m=+785.556912172" Nov 01 16:00:45.755474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:45.755440 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:51.019408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.019368 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg"] Nov 01 16:00:51.023207 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.023179 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.031404 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.031373 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"distinct-w5-164db6-kserve-self-signed-certs\"" Nov 01 16:00:51.035922 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.035890 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg"] Nov 01 16:00:51.164796 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.164752 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-dshm\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.165186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.165161 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624d676d-43a4-4cd8-8200-33a705076290-tls-certs\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.165429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.165412 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-home\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.165579 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.165563 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-tmp-dir\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.165760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.165745 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-model-cache\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.165957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.165942 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z9lr\" (UniqueName: \"kubernetes.io/projected/624d676d-43a4-4cd8-8200-33a705076290-kube-api-access-7z9lr\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.267411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.267372 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-model-cache\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.267639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.267452 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z9lr\" (UniqueName: \"kubernetes.io/projected/624d676d-43a4-4cd8-8200-33a705076290-kube-api-access-7z9lr\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.267639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.267537 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-dshm\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.267639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.267597 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624d676d-43a4-4cd8-8200-33a705076290-tls-certs\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.267841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.267673 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-home\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.267913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.267894 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-model-cache\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.268124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.268062 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-home\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.268262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.268205 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-tmp-dir\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.268375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.267708 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-tmp-dir\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.270061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.269991 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-dshm\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.270737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.270715 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624d676d-43a4-4cd8-8200-33a705076290-tls-certs\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.276269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.276244 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z9lr\" (UniqueName: \"kubernetes.io/projected/624d676d-43a4-4cd8-8200-33a705076290-kube-api-access-7z9lr\") pod \"distinct-w5-164db6-kserve-d94795f-zjdkg\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.340988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.340955 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:51.628917 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:51.628888 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg"] Nov 01 16:00:51.629749 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:00:51.629721 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod624d676d_43a4_4cd8_8200_33a705076290.slice/crio-061626ccdacb87f924437e98ca3251311a156628ef61572c8779296d2e42ab16 WatchSource:0}: Error finding container 061626ccdacb87f924437e98ca3251311a156628ef61572c8779296d2e42ab16: Status 404 returned error can't find the container with id 061626ccdacb87f924437e98ca3251311a156628ef61572c8779296d2e42ab16 Nov 01 16:00:52.449675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:52.449623 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" event={"ID":"624d676d-43a4-4cd8-8200-33a705076290","Type":"ContainerStarted","Data":"3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617"} Nov 01 16:00:52.450141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:52.449683 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" event={"ID":"624d676d-43a4-4cd8-8200-33a705076290","Type":"ContainerStarted","Data":"061626ccdacb87f924437e98ca3251311a156628ef61572c8779296d2e42ab16"} Nov 01 16:00:52.468973 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:52.468909 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podStartSLOduration=1.468890631 podStartE2EDuration="1.468890631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:52.466246381 +0000 UTC m=+792.610701035" watchObservedRunningTime="2026-11-01 16:00:52.468890631 +0000 UTC m=+792.613345275" Nov 01 16:00:53.342110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:53.342076 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:00:54.755852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:54.755818 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:54.769314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:54.769280 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:55.482691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:55.482662 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:00:58.341075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.338159 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl"] Nov 01 16:00:58.346394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.346364 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.347660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.347630 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl"] Nov 01 16:00:58.355451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.355420 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-43fac8-openshift-default-dockercfg-4dq2m\"" Nov 01 16:00:58.515307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.515267 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf1230-7fdf-4775-b710-758043bf0a79-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.523586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.523541 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.523912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.523888 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.524534 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.524511 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.532170 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.532142 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.532440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.532420 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.532628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.532611 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.532802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.532785 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz52z\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-kube-api-access-rz52z\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.533694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.533661 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.639840 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz52z\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-kube-api-access-rz52z\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.639927 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.639999 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf1230-7fdf-4775-b710-758043bf0a79-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.640091 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.640152 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.640179 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.640211 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.640282 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.647109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.640325 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.696473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.672145 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.696473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.672260 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.696473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.673511 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf1230-7fdf-4775-b710-758043bf0a79-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.696473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.674013 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.732164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.732088 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.779208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.779165 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz52z\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-kube-api-access-rz52z\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.780520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.780460 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.786069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.784120 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.799059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.794220 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:58.826576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:58.826286 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:00:59.331270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.331175 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp"] Nov 01 16:00:59.339240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.338202 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp"] Nov 01 16:00:59.339388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.338305 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.405818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.405782 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.405838 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.405877 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.405925 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.405958 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ngq\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-kube-api-access-h8ngq\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.406005 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.406135 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.406197 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.406333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.406291 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.448015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.447941 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl"] Nov 01 16:00:59.452676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.452386 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.452676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.452486 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.452676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.452532 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.507213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507168 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507237 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507301 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507329 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507362 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507389 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507418 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507441 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ngq\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-kube-api-access-h8ngq\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.507797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507468 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.509431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.507872 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.509431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.508251 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.509431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.508529 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.509431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.508906 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.510067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.510029 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.521107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.519285 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.523901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.522472 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.524275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.524225 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ngq\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-kube-api-access-h8ngq\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.528753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.528726 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.620742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.620631 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" event={"ID":"bbbf1230-7fdf-4775-b710-758043bf0a79","Type":"ContainerStarted","Data":"e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045"} Nov 01 16:00:59.620742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.620683 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" event={"ID":"bbbf1230-7fdf-4775-b710-758043bf0a79","Type":"ContainerStarted","Data":"83c2d6d8bf5e47856a6a626c4424465da5a285251cb6e5d06c8145c84a5b138a"} Nov 01 16:00:59.644597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.644540 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podStartSLOduration=1.644523962 podStartE2EDuration="1.644523962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:59.639857074 +0000 UTC m=+799.784311715" watchObservedRunningTime="2026-11-01 16:00:59.644523962 +0000 UTC m=+799.788978603" Nov 01 16:00:59.678629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.678588 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:00:59.827767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:00:59.827723 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:01:00.008784 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:00.008752 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae43db7_d6c2_40f7_ad3a_a871b1703e8d.slice/crio-ae1c83cb6b95cc70efc84e1386642b9e8251a4fabf1e43697488a93ab45617e8 WatchSource:0}: Error finding container ae1c83cb6b95cc70efc84e1386642b9e8251a4fabf1e43697488a93ab45617e8: Status 404 returned error can't find the container with id ae1c83cb6b95cc70efc84e1386642b9e8251a4fabf1e43697488a93ab45617e8 Nov 01 16:01:00.010867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.010841 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp"] Nov 01 16:01:00.011761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.011726 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.011859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.011788 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.011859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.011815 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.630752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.630708 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" event={"ID":"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d","Type":"ContainerStarted","Data":"b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf"} Nov 01 16:01:00.631184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.630818 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" event={"ID":"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d","Type":"ContainerStarted","Data":"ae1c83cb6b95cc70efc84e1386642b9e8251a4fabf1e43697488a93ab45617e8"} Nov 01 16:01:00.680178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.680138 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:01:00.680496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.680438 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podStartSLOduration=1.68042015 podStartE2EDuration="1.68042015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:00.671609329 +0000 UTC m=+800.816063970" watchObservedRunningTime="2026-11-01 16:01:00.68042015 +0000 UTC m=+800.824874793" Nov 01 16:01:00.828154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.828119 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 16:01:00.828328 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:00.828176 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 16:01:01.341431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:01.341395 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:01:01.412981 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:01.412945 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:01:01.685099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:01.684981 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 16:01:01.685710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:01.685659 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 16:01:01.816064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:01.810460 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:01:01.827750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:01.827712 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 16:01:01.827944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:01.827794 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 16:01:02.684606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:02.684143 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 16:01:02.684606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:02.684209 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 16:01:02.827220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:02.827181 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 16:01:02.827620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:02.827255 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 16:01:03.684909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:03.684432 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 16:01:03.684909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:03.684497 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 16:01:03.827835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:03.827795 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 16:01:03.828263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:03.827853 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 16:01:04.682061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:04.681538 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 16:01:04.682061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:04.681609 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 16:01:04.827210 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:04.827174 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 16:01:04.827393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:04.827234 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 16:01:05.682066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:05.681426 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 16:01:05.682066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:05.681510 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 16:01:05.827682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:05.827633 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 16:01:05.827851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:05.827704 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 16:01:06.679580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:06.679516 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 16:01:06.679908 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:06.679584 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 16:01:07.119691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:07.119650 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:01:07.120637 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:07.120605 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:01:07.121626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:07.121592 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:01:07.683402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:07.683371 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:01:07.683746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:07.683723 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:01:07.684468 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:07.684442 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:01:08.413526 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.413491 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6"] Nov 01 16:01:08.419863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.419837 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.429191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.429166 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"distinct-2-w2-6b0e0f-kserve-self-signed-certs\"" Nov 01 16:01:08.430872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.430840 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6"] Nov 01 16:01:08.517017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.516923 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bsx\" (UniqueName: \"kubernetes.io/projected/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-kube-api-access-89bsx\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.517232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.517155 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tls-certs\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.517390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.517367 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tmp-dir\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.517538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.517416 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-home\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.517538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.517479 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-model-cache\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.517538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.517514 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-dshm\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.618704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.618663 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bsx\" (UniqueName: \"kubernetes.io/projected/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-kube-api-access-89bsx\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.618887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.618784 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tls-certs\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.618887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.618836 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tmp-dir\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.618887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.618858 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-home\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.619075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.618890 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-model-cache\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.619075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.618914 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-dshm\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.619337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.619311 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tmp-dir\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.619442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.619361 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-home\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.619591 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.619570 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-model-cache\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.622287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.622265 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-dshm\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.622528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.622511 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tls-certs\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.629736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.629707 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bsx\" (UniqueName: \"kubernetes.io/projected/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-kube-api-access-89bsx\") pod \"distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:08.738133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:08.738097 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:09.029556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:09.029522 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6"] Nov 01 16:01:09.035580 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:09.035539 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6faee2de_c73d_486b_9d70_3f5dfcc8d8a6.slice/crio-505b594a900ff9fb62390e9aee6dabd16a2e8c6e076825a9812b53d61cf7d35a WatchSource:0}: Error finding container 505b594a900ff9fb62390e9aee6dabd16a2e8c6e076825a9812b53d61cf7d35a: Status 404 returned error can't find the container with id 505b594a900ff9fb62390e9aee6dabd16a2e8c6e076825a9812b53d61cf7d35a Nov 01 16:01:09.765685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:09.765644 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" event={"ID":"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6","Type":"ContainerStarted","Data":"f761c8391753ff28c3edd55997bee44482d57c2d3862a3c574768e316b09e87f"} Nov 01 16:01:09.766156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:09.765697 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" event={"ID":"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6","Type":"ContainerStarted","Data":"505b594a900ff9fb62390e9aee6dabd16a2e8c6e076825a9812b53d61cf7d35a"} Nov 01 16:01:09.785193 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:09.785145 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podStartSLOduration=1.785132843 podStartE2EDuration="1.785132843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:09.781803621 +0000 UTC m=+809.926258265" watchObservedRunningTime="2026-11-01 16:01:09.785132843 +0000 UTC m=+809.929587484" Nov 01 16:01:10.739304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:10.739270 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:10.866427 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:01:10.866388 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:01:13.556991 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.556955 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b"] Nov 01 16:01:13.561950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.561916 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.582151 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.582123 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b"] Nov 01 16:01:13.682946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.682907 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdcc\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-kube-api-access-vjdcc\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.682974 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.683097 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.683140 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683302 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.683160 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683302 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.683274 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57cb508-00c4-4520-a10e-f84d608df2ce-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.683308 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.683405 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.683485 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.683446 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.706173 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.706141 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r"] Nov 01 16:01:13.711446 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.711414 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.757107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.757070 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r"] Nov 01 16:01:13.784994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.784943 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785010 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785150 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdcc\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-kube-api-access-vjdcc\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785199 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785232 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785270 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785295 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785362 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57cb508-00c4-4520-a10e-f84d608df2ce-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.785425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.785391 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.786753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.786410 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.786753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.786685 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57cb508-00c4-4520-a10e-f84d608df2ce-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.786959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.786862 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.787285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.787262 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.787557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.787518 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.788805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.788785 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.788959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.788806 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.795505 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.795477 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdcc\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-kube-api-access-vjdcc\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.804615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.804528 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.886454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.886349 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:13.886945 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.886900 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887050 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887091 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887129 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887163 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887223 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9dc\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-kube-api-access-kl9dc\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887355 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887384 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.887497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.887444 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989108 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989176 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989213 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989279 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9dc\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-kube-api-access-kl9dc\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989413 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989444 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989505 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989580 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989647 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.989679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989652 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.990350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989959 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.990350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.989991 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.990350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.990085 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.990350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.990295 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.992611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.992581 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.993321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.993273 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.997721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.997683 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9dc\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-kube-api-access-kl9dc\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:13.998157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:13.998134 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:14.047086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.047020 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:14.085590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.082200 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z"] Nov 01 16:01:14.089511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.089295 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.097077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.096991 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z"] Nov 01 16:01:14.181816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.181787 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b"] Nov 01 16:01:14.182342 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:14.182318 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57cb508_00c4_4520_a10e_f84d608df2ce.slice/crio-6d2a10652eaaef1ed54678ec20622b1b4b0991af7be91f00dea2f712c3f73337 WatchSource:0}: Error finding container 6d2a10652eaaef1ed54678ec20622b1b4b0991af7be91f00dea2f712c3f73337: Status 404 returned error can't find the container with id 6d2a10652eaaef1ed54678ec20622b1b4b0991af7be91f00dea2f712c3f73337 Nov 01 16:01:14.185134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.185104 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.185227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.185180 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.185296 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.185223 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.192153 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192128 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192190 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192245 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192400 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192378 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192426 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192735 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192764 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkf9l\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-kube-api-access-lkf9l\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192805 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.192951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.192937 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294473 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294570 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294610 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294698 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294734 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294794 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294820 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkf9l\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-kube-api-access-lkf9l\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294871 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.294980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294947 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.295190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294953 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.295190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.294991 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.295309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.295263 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.295361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.295307 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.296196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.296168 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.297629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.297605 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.297817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.297795 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.304878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.304794 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkf9l\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-kube-api-access-lkf9l\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.304878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.304833 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.373443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.373407 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r"] Nov 01 16:01:14.373882 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:14.373837 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07ef0c7_b67d_4849_a0f1_b4b84aa022e3.slice/crio-fe8c25258ace5cf5e3c201a7f64ad614b5e099e5de621cdffd4699eb426197a6 WatchSource:0}: Error finding container fe8c25258ace5cf5e3c201a7f64ad614b5e099e5de621cdffd4699eb426197a6: Status 404 returned error can't find the container with id fe8c25258ace5cf5e3c201a7f64ad614b5e099e5de621cdffd4699eb426197a6 Nov 01 16:01:14.376556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.376524 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.376671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.376585 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.376671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.376617 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.411988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.411965 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:14.707502 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:14.707464 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542c63ce_b126_4f1d_8eaf_d5ca3e087df6.slice/crio-1599bfb6f8d2af965a35d823a6d1950dded9e25eb71a23267c2bd204af2bd1e9 WatchSource:0}: Error finding container 1599bfb6f8d2af965a35d823a6d1950dded9e25eb71a23267c2bd204af2bd1e9: Status 404 returned error can't find the container with id 1599bfb6f8d2af965a35d823a6d1950dded9e25eb71a23267c2bd204af2bd1e9 Nov 01 16:01:14.708837 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.708810 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z"] Nov 01 16:01:14.710352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.710322 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.710443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.710394 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.710517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.710440 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.814358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.813293 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" event={"ID":"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3","Type":"ContainerStarted","Data":"3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e"} Nov 01 16:01:14.814358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.813360 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" event={"ID":"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3","Type":"ContainerStarted","Data":"fe8c25258ace5cf5e3c201a7f64ad614b5e099e5de621cdffd4699eb426197a6"} Nov 01 16:01:14.817135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.817105 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" event={"ID":"b57cb508-00c4-4520-a10e-f84d608df2ce","Type":"ContainerStarted","Data":"6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190"} Nov 01 16:01:14.817336 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.817298 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" event={"ID":"b57cb508-00c4-4520-a10e-f84d608df2ce","Type":"ContainerStarted","Data":"6d2a10652eaaef1ed54678ec20622b1b4b0991af7be91f00dea2f712c3f73337"} Nov 01 16:01:14.820026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.819997 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" event={"ID":"542c63ce-b126-4f1d-8eaf-d5ca3e087df6","Type":"ContainerStarted","Data":"710013b9872a8d2bd0c5bc7b16b179feb9f053d3dd4cb7d81cc8ea3ba54e6793"} Nov 01 16:01:14.820164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.820086 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" event={"ID":"542c63ce-b126-4f1d-8eaf-d5ca3e087df6","Type":"ContainerStarted","Data":"1599bfb6f8d2af965a35d823a6d1950dded9e25eb71a23267c2bd204af2bd1e9"} Nov 01 16:01:14.834676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.834633 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podStartSLOduration=1.834620858 podStartE2EDuration="1.834620858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:14.833792696 +0000 UTC m=+814.978247358" watchObservedRunningTime="2026-11-01 16:01:14.834620858 +0000 UTC m=+814.979075499" Nov 01 16:01:14.854559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.854486 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podStartSLOduration=1.8544684569999998 podStartE2EDuration="1.854468457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:14.853516433 +0000 UTC m=+814.997971081" watchObservedRunningTime="2026-11-01 16:01:14.854468457 +0000 UTC m=+814.998923164" Nov 01 16:01:14.889087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.887414 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:14.890498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:14.890374 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podStartSLOduration=0.890348593 podStartE2EDuration="890.348593ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:14.888872004 +0000 UTC m=+815.033326642" watchObservedRunningTime="2026-11-01 16:01:14.890348593 +0000 UTC m=+815.034803299" Nov 01 16:01:15.047980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.047943 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:15.049657 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.049620 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:15.049803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.049685 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 16:01:15.412904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.412799 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:15.417067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.414972 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:15.417067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.415060 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:15.898062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.897660 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:15.898062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:15.897728 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 16:01:16.048195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:16.048156 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.048484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:16.048453 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 16:01:16.418065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:16.413505 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.418065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:16.413572 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:16.894493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:16.894452 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.894790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:16.894758 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 16:01:17.047922 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:17.047888 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.048297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:17.047940 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 16:01:17.412539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:17.412497 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.412862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:17.412831 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:17.891067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:17.888654 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.891067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:17.888720 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 16:01:18.048183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.048144 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.048592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.048211 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 16:01:18.414444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.414403 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.414722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.414694 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:18.762064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.759341 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:18.889663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.889628 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.890152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.890117 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 16:01:18.944575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:18.944528 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:19.047975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:19.047449 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.047975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:19.047511 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 16:01:19.146105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:19.146072 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:19.417195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:19.413555 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.417195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:19.413614 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:19.890065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:19.887089 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.890065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:19.887166 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 16:01:20.049084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:20.048053 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.049084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:20.048132 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 16:01:20.467301 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:20.467266 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.475870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:20.475410 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:20.890066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:20.889654 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.890066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:20.889726 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 16:01:21.075203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:21.075160 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:21.075503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:21.075466 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 01 16:01:21.414061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:21.413187 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:21.414061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:21.413270 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:21.890066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:21.889443 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:21.890066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:21.889509 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 01 16:01:22.418061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:22.417462 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:22.418061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:22.417533 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 01 16:01:22.949324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:22.948346 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.049876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.049267 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:23.049876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.049333 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.517063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.516668 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:01:23.547061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.546136 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.730189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.730131 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.804304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.804178 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:23.804618 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.804581 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.892187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.892144 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:23.892443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.892413 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.949514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.949465 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.952008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:23.951981 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:24.134369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.134278 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:24.136064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.136025 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:24.143935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.143905 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:01:24.161225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.161184 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:24.161513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.161478 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.193065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.183306 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:24.193065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.184617 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:24.199063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.195517 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:01:24.346457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.345376 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:01:24.356684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.356262 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:01:24.401122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.400307 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.521172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.521124 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.522017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.521992 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:01:24.524024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.523994 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.616060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.611142 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:01:24.616060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.611663 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:01:24.648432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.645478 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 01 16:01:24.952937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:24.951987 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 01 16:01:25.158064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.128999 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:25.237124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.231800 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:01:25.279064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.261010 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:25.279064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.272897 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.673124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.672030 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:01:25.745215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.745160 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:25.770492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.770456 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:01:25.955115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.953314 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:25.993160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:25.993124 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:26.007386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:26.007342 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:01:26.117026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:26.116829 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:01:26.122134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:26.122106 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:01:28.665723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.665682 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l"] Nov 01 16:01:28.670624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.670597 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.712308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.712270 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"default-dockercfg-bxf27\"" Nov 01 16:01:28.712467 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.712376 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"openshift-service-ca.crt\"" Nov 01 16:01:28.712577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.712542 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"kube-root-ca.crt\"" Nov 01 16:01:28.712577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.712546 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"distinct-2-w3-bd831a-kserve-self-signed-certs\"" Nov 01 16:01:28.713244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.711414 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l"] Nov 01 16:01:28.840061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.835909 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnmw\" (UniqueName: \"kubernetes.io/projected/be0b5d67-06b1-4084-9754-3778e9a728d8-kube-api-access-zwnmw\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.840061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.836186 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-model-cache\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.840061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.836234 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0b5d67-06b1-4084-9754-3778e9a728d8-tls-certs\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.840061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.836281 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-home\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.840061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.836305 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-tmp-dir\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.840061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.836367 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-dshm\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937240 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-model-cache\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937283 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0b5d67-06b1-4084-9754-3778e9a728d8-tls-certs\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937319 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-home\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937344 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-tmp-dir\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937379 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-dshm\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937470 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnmw\" (UniqueName: \"kubernetes.io/projected/be0b5d67-06b1-4084-9754-3778e9a728d8-kube-api-access-zwnmw\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937771 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937747 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-model-cache\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.937929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.937903 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-tmp-dir\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.938065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.938025 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-home\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.939930 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.939904 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-dshm\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.940809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.940788 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0b5d67-06b1-4084-9754-3778e9a728d8-tls-certs\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:28.949577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:28.949551 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnmw\" (UniqueName: \"kubernetes.io/projected/be0b5d67-06b1-4084-9754-3778e9a728d8-kube-api-access-zwnmw\") pod \"distinct-2-w3-bd831a-kserve-7555456544-48p6l\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:29.017557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:29.017509 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:29.308136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:29.308102 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l"] Nov 01 16:01:29.308404 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:29.308381 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe0b5d67_06b1_4084_9754_3778e9a728d8.slice/crio-cf0a0ef9befc653ca1d8203daaf4b9d2279a4ec6d4cc53495af9b156c0816a25 WatchSource:0}: Error finding container cf0a0ef9befc653ca1d8203daaf4b9d2279a4ec6d4cc53495af9b156c0816a25: Status 404 returned error can't find the container with id cf0a0ef9befc653ca1d8203daaf4b9d2279a4ec6d4cc53495af9b156c0816a25 Nov 01 16:01:29.544518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:29.544482 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" event={"ID":"be0b5d67-06b1-4084-9754-3778e9a728d8","Type":"ContainerStarted","Data":"0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6"} Nov 01 16:01:29.544518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:29.544523 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" event={"ID":"be0b5d67-06b1-4084-9754-3778e9a728d8","Type":"ContainerStarted","Data":"cf0a0ef9befc653ca1d8203daaf4b9d2279a4ec6d4cc53495af9b156c0816a25"} Nov 01 16:01:29.563662 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:29.563536 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podStartSLOduration=1.56351889 podStartE2EDuration="1.56351889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:29.560227189 +0000 UTC m=+829.704681832" watchObservedRunningTime="2026-11-01 16:01:29.56351889 +0000 UTC m=+829.707973532" Nov 01 16:01:30.018783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:30.018694 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:36.503393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.503355 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97"] Nov 01 16:01:36.509762 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.509733 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.517075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.517013 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97"] Nov 01 16:01:36.521527 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.521456 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-164db6\"/\"distinct-2-w5-164db6-kserve-self-signed-certs\"" Nov 01 16:01:36.547074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.544731 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-model-cache\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.547074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.544815 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-dshm\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.547074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.544843 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcsn\" (UniqueName: \"kubernetes.io/projected/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-kube-api-access-tlcsn\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.547074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.544952 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-home\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.547074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.545001 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tmp-dir\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.547074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.545057 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tls-certs\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.651989 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-home\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.652095 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tmp-dir\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.652171 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tls-certs\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.652263 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-model-cache\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.652356 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-dshm\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.652383 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcsn\" (UniqueName: \"kubernetes.io/projected/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-kube-api-access-tlcsn\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.653563 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-home\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.654337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.653847 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tmp-dir\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.665617 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.656392 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-model-cache\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.665617 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.668375 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-dshm\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.665617 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.678020 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcsn\" (UniqueName: \"kubernetes.io/projected/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-kube-api-access-tlcsn\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.732152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.726417 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tls-certs\") pod \"distinct-2-w5-164db6-kserve-64c769844d-cnz97\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:36.835210 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:36.835171 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:37.128942 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:37.128915 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaba15e06_9a58_43e7_94c7_8d8aba22d4e5.slice/crio-f0c9be7d7ed498b3a68205a4bb67c471375b62f803f7db4e55a3ff3c360fb23b WatchSource:0}: Error finding container f0c9be7d7ed498b3a68205a4bb67c471375b62f803f7db4e55a3ff3c360fb23b: Status 404 returned error can't find the container with id f0c9be7d7ed498b3a68205a4bb67c471375b62f803f7db4e55a3ff3c360fb23b Nov 01 16:01:37.129676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:37.129627 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97"] Nov 01 16:01:37.625375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:37.625328 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" event={"ID":"aba15e06-9a58-43e7-94c7-8d8aba22d4e5","Type":"ContainerStarted","Data":"01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad"} Nov 01 16:01:37.625981 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:37.625386 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" event={"ID":"aba15e06-9a58-43e7-94c7-8d8aba22d4e5","Type":"ContainerStarted","Data":"f0c9be7d7ed498b3a68205a4bb67c471375b62f803f7db4e55a3ff3c360fb23b"} Nov 01 16:01:37.645304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:37.645215 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podStartSLOduration=1.645192907 podStartE2EDuration="1.645192907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:37.643352981 +0000 UTC m=+837.787807623" watchObservedRunningTime="2026-11-01 16:01:37.645192907 +0000 UTC m=+837.789647568" Nov 01 16:01:37.835526 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:37.835486 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:38.422977 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.422942 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68"] Nov 01 16:01:38.430242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.428737 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.433111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.433078 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68"] Nov 01 16:01:38.438550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.438471 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"distinct-2-w1-dfff59-kserve-self-signed-certs\"" Nov 01 16:01:38.494993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.493923 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-tmp-dir\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.494993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.494106 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-dshm\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.494993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.494251 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a13fd4-ab3e-42a5-872e-8eccecef961c-tls-certs\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.494993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.494289 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-model-cache\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.494993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.494389 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcd9c\" (UniqueName: \"kubernetes.io/projected/99a13fd4-ab3e-42a5-872e-8eccecef961c-kube-api-access-xcd9c\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.494993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.494514 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-home\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.602956 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-dshm\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.603097 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a13fd4-ab3e-42a5-872e-8eccecef961c-tls-certs\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.603122 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-model-cache\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.603197 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcd9c\" (UniqueName: \"kubernetes.io/projected/99a13fd4-ab3e-42a5-872e-8eccecef961c-kube-api-access-xcd9c\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.603284 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-home\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.603352 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-tmp-dir\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.603812 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-tmp-dir\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.603838 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-model-cache\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.604080 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-home\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606643 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.606271 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-dshm\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.606705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.606662 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a13fd4-ab3e-42a5-872e-8eccecef961c-tls-certs\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.613757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.613728 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcd9c\" (UniqueName: \"kubernetes.io/projected/99a13fd4-ab3e-42a5-872e-8eccecef961c-kube-api-access-xcd9c\") pod \"distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:38.748776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:38.748734 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:39.018228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.018145 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:39.031696 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.031668 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:39.039238 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:39.039214 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99a13fd4_ab3e_42a5_872e_8eccecef961c.slice/crio-cbf8f41b5504acb6171996b5f3656728b788e3bce91efdb58fc7d64b011b941a WatchSource:0}: Error finding container cbf8f41b5504acb6171996b5f3656728b788e3bce91efdb58fc7d64b011b941a: Status 404 returned error can't find the container with id cbf8f41b5504acb6171996b5f3656728b788e3bce91efdb58fc7d64b011b941a Nov 01 16:01:39.039354 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.039238 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68"] Nov 01 16:01:39.643845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.643802 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" event={"ID":"99a13fd4-ab3e-42a5-872e-8eccecef961c","Type":"ContainerStarted","Data":"5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b"} Nov 01 16:01:39.643845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.643853 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" event={"ID":"99a13fd4-ab3e-42a5-872e-8eccecef961c","Type":"ContainerStarted","Data":"cbf8f41b5504acb6171996b5f3656728b788e3bce91efdb58fc7d64b011b941a"} Nov 01 16:01:39.655891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.655848 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:39.662016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.661972 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podStartSLOduration=1.661959094 podStartE2EDuration="1.661959094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:39.659591285 +0000 UTC m=+839.804045923" watchObservedRunningTime="2026-11-01 16:01:39.661959094 +0000 UTC m=+839.806413789" Nov 01 16:01:39.749504 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:39.749476 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:40.869371 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:01:40.869335 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:01:43.905317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.905282 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt"] Nov 01 16:01:43.916065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.911701 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.927090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.924478 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt"] Nov 01 16:01:43.977872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.977837 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.977900 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.977988 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.978033 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a71a41-49b4-4557-8cea-e9f9f01f7709-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.978134 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ggg5\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-kube-api-access-8ggg5\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.978224 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.978245 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.978259 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:43.978465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:43.978346 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.026509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.026472 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7"] Nov 01 16:01:44.030779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.030759 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.045190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.045158 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7"] Nov 01 16:01:44.058711 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.058233 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s"] Nov 01 16:01:44.065332 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.063622 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.080319 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080281 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a71a41-49b4-4557-8cea-e9f9f01f7709-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.080492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080339 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.080492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080370 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ggg5\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-kube-api-access-8ggg5\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.080492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080442 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.080492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080470 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.080721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080561 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.080721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080589 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.080721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080606 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.080721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080642 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.080721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080675 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.080721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080705 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.081022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080732 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.081022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080748 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.081022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080789 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.081022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080813 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.081022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080833 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxht\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-kube-api-access-hcxht\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.081022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080889 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.081022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.080910 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.081413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.081228 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a71a41-49b4-4557-8cea-e9f9f01f7709-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.081841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.081693 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.081841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.081799 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.082069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.081963 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.082322 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.082287 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.085748 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.085163 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s"] Nov 01 16:01:44.091511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.091483 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.093086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.093024 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.098140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.098113 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ggg5\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-kube-api-access-8ggg5\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.105215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.105191 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.182569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182473 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.182569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182515 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.182569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182541 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.182866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182647 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.182866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182712 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.182866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182796 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.182866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182832 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxht\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-kube-api-access-hcxht\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.183126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182893 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.183126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182927 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k52f\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-kube-api-access-7k52f\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.183126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.182960 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.183126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183076 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.183126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183107 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.183371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183162 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.183371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183216 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.183371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183263 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.183371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183288 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.183585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183372 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.183585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183427 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.183694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.183634 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.184304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.184275 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.184304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.184290 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.184466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.184360 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.184538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.184509 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.188128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.186424 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.189865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.189575 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.195022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.194979 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxht\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-kube-api-access-hcxht\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.203803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.203144 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.234795 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.234762 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:44.246158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.246129 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2"] Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.283273 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2"] Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.283410 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.285876 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.285944 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.285992 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.286088 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.286168 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.286205 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.286241 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.286382 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.286937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.286407 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k52f\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-kube-api-access-7k52f\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.287347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.287309 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.287702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.287676 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.291013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.289280 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.291367 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.291343 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.291659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.291604 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.293324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.292592 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.295707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.295660 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k52f\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-kube-api-access-7k52f\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.296121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.296072 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.297104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.296651 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.347349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.347314 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388243 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388314 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388365 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388439 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhmc\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-kube-api-access-zzhmc\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388478 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388510 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388578 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388603 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.391028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.388631 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.396254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.395686 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:44.490648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490414 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.490648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490451 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.490648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490472 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.490648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490613 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.490648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490643 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.491084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490681 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.491084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490730 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhmc\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-kube-api-access-zzhmc\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.491084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490756 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.491084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.490778 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.491300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.491262 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.491961 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.491934 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.492779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.492501 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.494088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.493402 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.494088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.494020 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.496901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.496855 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.498350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.498201 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.499907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.499880 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.500418 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.500398 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhmc\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-kube-api-access-zzhmc\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.554805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.554775 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt"] Nov 01 16:01:44.555525 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:44.555491 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a71a41_49b4_4557_8cea_e9f9f01f7709.slice/crio-51bcac5c2e5937a8626c2e0c6c63f4fc6c2cd5fcefcc9ab98e4e5601a685b984 WatchSource:0}: Error finding container 51bcac5c2e5937a8626c2e0c6c63f4fc6c2cd5fcefcc9ab98e4e5601a685b984: Status 404 returned error can't find the container with id 51bcac5c2e5937a8626c2e0c6c63f4fc6c2cd5fcefcc9ab98e4e5601a685b984 Nov 01 16:01:44.558289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.558258 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.558398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.558328 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.558398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.558358 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.618782 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.618750 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:44.657742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.657596 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7"] Nov 01 16:01:44.674333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.666710 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.674333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.666780 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.674333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.666813 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.713746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.713168 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s"] Nov 01 16:01:44.718867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.718777 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" event={"ID":"0dbd1cae-eb9b-4e44-aeea-731867bb217f","Type":"ContainerStarted","Data":"e15ed581cfa417f3c6d961c1a30b3b9eb2b67f415f3e6ba50111286e1afbb7e5"} Nov 01 16:01:44.722285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.721208 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" event={"ID":"16a71a41-49b4-4557-8cea-e9f9f01f7709","Type":"ContainerStarted","Data":"bc227bd9ac7841ffb8169336a9457a922f3c253bc346afd01e06f8b4d4f8556a"} Nov 01 16:01:44.722285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.721256 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" event={"ID":"16a71a41-49b4-4557-8cea-e9f9f01f7709","Type":"ContainerStarted","Data":"51bcac5c2e5937a8626c2e0c6c63f4fc6c2cd5fcefcc9ab98e4e5601a685b984"} Nov 01 16:01:44.724685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.723995 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.724685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.724093 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.724685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.724138 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.765265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.764517 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podStartSLOduration=1.7644961019999998 podStartE2EDuration="1.764496102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:44.754526744 +0000 UTC m=+844.898981387" watchObservedRunningTime="2026-11-01 16:01:44.764496102 +0000 UTC m=+844.908950744" Nov 01 16:01:44.945907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.945166 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2"] Nov 01 16:01:44.952818 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:01:44.952729 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7150a085_b0d4_46a0_8a41_cc510b4ba44e.slice/crio-a44a084253562c5bddb546984ef8164633103901f826298dbaa4a2aad7ceaef8 WatchSource:0}: Error finding container a44a084253562c5bddb546984ef8164633103901f826298dbaa4a2aad7ceaef8: Status 404 returned error can't find the container with id a44a084253562c5bddb546984ef8164633103901f826298dbaa4a2aad7ceaef8 Nov 01 16:01:44.957181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.956194 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.957181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.956284 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.957181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:44.956326 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:45.235187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.235095 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:45.236784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.236745 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:45.236931 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.236808 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:45.770890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.770845 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" event={"ID":"0dbd1cae-eb9b-4e44-aeea-731867bb217f","Type":"ContainerStarted","Data":"95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4"} Nov 01 16:01:45.828062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.823243 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" event={"ID":"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df","Type":"ContainerStarted","Data":"2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c"} Nov 01 16:01:45.828062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.823314 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" event={"ID":"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df","Type":"ContainerStarted","Data":"dd4b952b6c8e04d4a7bfebdd88f9626a28eb9b95d5a9dc35453944b47db10676"} Nov 01 16:01:45.850077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.841704 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podStartSLOduration=1.84168343 podStartE2EDuration="1.84168343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:45.820569639 +0000 UTC m=+845.965024292" watchObservedRunningTime="2026-11-01 16:01:45.84168343 +0000 UTC m=+845.986138076" Nov 01 16:01:45.933065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:45.896359 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podStartSLOduration=1.8963386679999998 podStartE2EDuration="1.896338668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:45.89243158 +0000 UTC m=+846.036886234" watchObservedRunningTime="2026-11-01 16:01:45.896338668 +0000 UTC m=+846.040793309" Nov 01 16:01:46.013932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.000545 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" event={"ID":"7150a085-b0d4-46a0-8a41-cc510b4ba44e","Type":"ContainerStarted","Data":"1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34"} Nov 01 16:01:46.013932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.000620 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" event={"ID":"7150a085-b0d4-46a0-8a41-cc510b4ba44e","Type":"ContainerStarted","Data":"a44a084253562c5bddb546984ef8164633103901f826298dbaa4a2aad7ceaef8"} Nov 01 16:01:46.238553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.238518 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.238797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.238768 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:46.375104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.375070 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:46.375466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.375445 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.376206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.376176 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:46.418437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.417176 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:46.421290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.421169 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.421290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.421226 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:46.641250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.641154 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:46.641923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.641499 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.641923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.641650 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:46.838192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:46.838164 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:47.250582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.250543 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.284069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.284008 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:47.301560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.300830 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.348592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.348550 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.362074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.349860 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.362376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.362341 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:47.396647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.396606 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.405162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.405118 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:47.415066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.398395 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:01:47.477064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.476236 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.625553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.623498 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.625553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.623569 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:47.677593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.677544 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.708192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.708139 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.842195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:47.842145 4911 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.187060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.185391 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.249082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.248507 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.249082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.248572 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:48.275179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.265391 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.450290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.450179 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.467183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.467131 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:48.525539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.524173 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.525539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.524577 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:01:48.635067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.606783 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podStartSLOduration=4.606758774 podStartE2EDuration="4.606758774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:46.03112702 +0000 UTC m=+846.175581664" watchObservedRunningTime="2026-11-01 16:01:48.606758774 +0000 UTC m=+848.751213416" Nov 01 16:01:48.646087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.627993 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.646383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.646348 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:48.691234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.691191 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.692762 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.692734 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:48.693310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.692655 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.693849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.693828 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:48.769076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.764136 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:48.769564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.769373 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:48.769564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.769499 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:01:48.964090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:48.963309 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.121305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.121201 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:01:49.216285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.216232 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:01:49.224109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.224070 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:01:49.299499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.299460 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.537186 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.330177 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.539120 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.324851 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.428203 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.405922 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.542393 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.455674 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.543483 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:49.548064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.427976 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.600906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.428597 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:01:49.603060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.603015 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:01:49.644064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.643761 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.644064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.643947 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:49.724061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.708651 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.762063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.761210 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.799113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.798666 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:49.813064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.812464 4911 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.984725 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:49.984684 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:50.179251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.179138 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:01:50.200292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.200246 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:50.248065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.216808 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:01:50.248065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.245147 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.248065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.245221 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:50.439123 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.438713 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs"] Nov 01 16:01:50.445085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.440795 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.445434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.445385 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:50.448004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.447980 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.448209 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.448185 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:50.532577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.515936 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs"] Nov 01 16:01:50.532980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.532955 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:01:50.540905 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.540871 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:01:50.541451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.541421 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:01:50.541866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.541846 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:01:50.542470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.542436 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:50.544019 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.543994 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:01:50.545055 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.544194 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:01:50.559245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.544260 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.609148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.583992 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-model-cache\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.621117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.609553 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b523b317-2c74-4159-a618-ce4a6d9635c5-tls-certs\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.622211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.622177 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-tmp-dir\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.622476 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.622457 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tq75\" (UniqueName: \"kubernetes.io/projected/b523b317-2c74-4159-a618-ce4a6d9635c5-kube-api-access-5tq75\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.622664 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.622647 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-home\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.622853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.622839 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-dshm\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.632149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.632108 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:50.658067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.632669 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:01:50.660032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.651111 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.660293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.660266 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:50.717349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.652470 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"kube-root-ca.crt\"" Nov 01 16:01:50.717349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.652326 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"default-dockercfg-lhz8h\"" Nov 01 16:01:50.717349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.703812 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"openshift-service-ca.crt\"" Nov 01 16:01:50.717349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.713867 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:01:50.748062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.742007 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-model-cache\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.786682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.784339 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b523b317-2c74-4159-a618-ce4a6d9635c5-tls-certs\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.786682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.784411 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-tmp-dir\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.786682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.784480 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tq75\" (UniqueName: \"kubernetes.io/projected/b523b317-2c74-4159-a618-ce4a6d9635c5-kube-api-access-5tq75\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.786682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.784573 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-home\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.786682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.784657 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-dshm\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.830117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.812694 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:50.830117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.747063 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-model-cache\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.830117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.757258 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"distinct-2-w0-4a083d-kserve-self-signed-certs\"" Nov 01 16:01:50.843154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.843121 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-home\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.843584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.843561 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-dshm\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:50.909113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.908687 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-tmp-dir\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:51.012125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:50.978751 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tq75\" (UniqueName: \"kubernetes.io/projected/b523b317-2c74-4159-a618-ce4a6d9635c5-kube-api-access-5tq75\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:51.199061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.191349 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.252774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.219301 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b523b317-2c74-4159-a618-ce4a6d9635c5-tls-certs\") pod \"distinct-2-w0-4a083d-kserve-55844486cd-679zs\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:51.252774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.220170 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.268295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.268168 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.285063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.272108 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:51.356065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.348068 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.356065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.348128 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:51.396598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.396567 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:01:51.416894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.398620 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.417212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.417182 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:51.465473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.465432 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.480325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.480284 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:51.620097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.619991 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.620344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.620315 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:51.620465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.620143 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:01:51.620937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.620111 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:01:51.621641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.620164 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:01:51.639518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.638278 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:01:51.718677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.718623 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.815561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:51.815514 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:52.030979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.030933 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.200200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.200152 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.224065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.222356 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.244755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.244692 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.244963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.244758 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:52.347136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.343144 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.353079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.351687 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.353079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.351748 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:52.398255 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.398211 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.413304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.401255 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.414773 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.414715 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:52.509196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.471673 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.509196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.472086 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:01:52.676120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.670165 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.734065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.733956 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:01:52.743075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.734322 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:01:52.795065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.791941 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.795065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.792070 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:01:52.806097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.802454 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.853206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.853157 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:52.853476 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.835129 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.853896 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:52.806344 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:01:53.215099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.212498 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.406760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.372417 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:53.434063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.407009 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:01:53.456075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.451082 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.456075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.452416 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:01:53.456075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.432282 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.456075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.454279 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:53.466461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.466367 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.492254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.473946 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:53.492254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.476458 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.572508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.572459 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.810064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.806837 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:01:53.844069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.842790 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.844069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.842869 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:53.871064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.857003 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Nov 01 16:01:53.871064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.857142 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:01:53.871064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.857902 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:01:53.871064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.865839 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs"] Nov 01 16:01:53.892700 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.892664 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:53.892902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.892874 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.942067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.938147 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.978362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.943073 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:01:53.982063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.977479 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.982636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:53.982609 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.147209 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.147156 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:54.148347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.148310 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:54.178160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.178097 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.219063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.215291 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.249127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.246020 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:01:54.249127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.249027 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.249387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.249115 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:54.376125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.350514 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.402198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.402148 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:54.405132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.405098 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.405384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.405355 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:54.455144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.455092 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.455848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.455801 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:01:54.513376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.513326 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.635252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.635140 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.635252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.635221 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:54.666052 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.666009 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:01:54.682405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.682367 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:01:54.683081 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.683029 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 01 16:01:54.868064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.855112 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.941141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.868794 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.941141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.869261 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.980534 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:54.980489 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:55.021456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.002299 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" event={"ID":"b523b317-2c74-4159-a618-ce4a6d9635c5","Type":"ContainerStarted","Data":"1e056b844078afc2719cd4e9315f6a1e584dff32b1475ad2aece240ebb95022e"} Nov 01 16:01:55.155732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.155689 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.218607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.218498 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.235480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.235437 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:55.235769 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.235740 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.242206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.242170 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.242964 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.242927 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:55.280066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.254428 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:55.280066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.254490 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.359104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.355718 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.363079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.359384 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:55.378252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.378198 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.400776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.400734 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.401111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.401077 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:55.407072 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:01:55.406988 4911 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb523b317_2c74_4159_a618_ce4a6d9635c5.slice/crio-1e056b844078afc2719cd4e9315f6a1e584dff32b1475ad2aece240ebb95022e\": RecentStats: unable to find data in memory cache]" Nov 01 16:01:55.455684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.455637 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.491377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.491338 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:01:55.532202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.532151 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.646258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.646216 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.646878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.646843 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:55.734066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.730158 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.774364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.774261 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.862188 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.862136 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.872065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.870972 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.872065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.871414 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:01:55.995064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:55.990591 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.168236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.167403 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:01:56.222072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.220130 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.222072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.220572 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 01 16:01:56.240187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.240134 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.247368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.247332 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.247660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.247630 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:56.455133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.426003 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.455461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.455424 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:56.455713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.448669 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.455713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.455670 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:56.456086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.456062 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.491293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.491246 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:01:56.544455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.544309 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.624831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.624779 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.625114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.625080 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 16:01:56.731261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.731143 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.749074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.732531 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:01:56.939800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.936614 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.939800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.937725 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:01:56.939800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.939129 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:01:56.960616 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:56.959908 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:01:57.172009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.171966 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:01:57.193486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.189183 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:57.222561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.222390 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.297372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.296479 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.297372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.296543 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 01 16:01:57.301964 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.299835 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:01:57.354866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.354832 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.355170 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.355129 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 01 16:01:57.379065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.375276 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:01:57.399451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.398263 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:01:57.399451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.399311 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:01:57.405061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.399679 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.405061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.399758 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 01 16:01:57.408602 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.408573 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:01:57.456499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.456461 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:01:57.505102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.501074 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:01:57.571061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.565776 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:01:57.582086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.575782 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:01:57.620001 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:57.617660 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:01:58.399061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.393399 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" event={"ID":"b523b317-2c74-4159-a618-ce4a6d9635c5","Type":"ContainerStarted","Data":"49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554"} Nov 01 16:01:58.434337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.434270 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podStartSLOduration=9.434248028 podStartE2EDuration="9.434248028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:58.425178068 +0000 UTC m=+858.569632734" watchObservedRunningTime="2026-11-01 16:01:58.434248028 +0000 UTC m=+858.578702670" Nov 01 16:01:58.486627 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.486590 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:01:58.492317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.491331 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:58.492847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.492817 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:58.511399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.511367 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:01:58.856146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.856114 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:58.926466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:58.925882 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:01:59.244064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.242256 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:01:59.244064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.243520 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:59.251061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.244895 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:59.251061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.245152 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:59.256061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.252134 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:59.290074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.288235 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:01:59.296724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.294015 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:01:59.361813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.357194 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:01:59.361813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:01:59.357281 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.097684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:00.095634 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:02:00.131063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:00.129537 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:02:01.484128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:01.484090 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:02:01.497768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:01.497740 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:02:01.547228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:01.547151 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:02:10.871861 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:02:10.871822 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:02:13.470762 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.470728 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc"] Nov 01 16:02:13.503796 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.502590 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.507135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.507095 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc"] Nov 01 16:02:13.555104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555026 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555353 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555555 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555656 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555698 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555728 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555836 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4zk\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-kube-api-access-fd4zk\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555882 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.555984 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.555940 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.657826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.657785 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.657995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.657862 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.657995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.657902 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658082 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4zk\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-kube-api-access-fd4zk\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658145 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658239 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658216 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658296 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658535 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658563 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658578 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.658961 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.658934 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.659070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.659003 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.659189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.659155 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.659724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.659690 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.660419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.660397 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.660742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.660721 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.669408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.669377 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn"] Nov 01 16:02:13.675404 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.675371 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.675558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.675529 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4zk\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-kube-api-access-fd4zk\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.678213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.678187 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.689517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.689388 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn"] Nov 01 16:02:13.759444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759397 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97794338-5b3b-447a-9aa3-221d194cdd77-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.759657 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759543 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.759657 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759649 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.759800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759721 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.759856 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759828 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8scwl\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-kube-api-access-8scwl\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.759923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759903 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.759967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759946 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97794338-5b3b-447a-9aa3-221d194cdd77-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.760017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.759979 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.760110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.760086 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.847357 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.847317 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:13.861489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.861443 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97794338-5b3b-447a-9aa3-221d194cdd77-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.861644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.861549 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.861644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.861619 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.861767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.861675 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97794338-5b3b-447a-9aa3-221d194cdd77-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.861767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.861740 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.861870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.861821 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.861924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.861890 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.862070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.862050 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8scwl\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-kube-api-access-8scwl\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.862198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.862117 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.862595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.862570 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.862861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.862839 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.863065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.863027 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.863855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.863829 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97794338-5b3b-447a-9aa3-221d194cdd77-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.864141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.864117 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.865220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.865200 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.865801 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.865777 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97794338-5b3b-447a-9aa3-221d194cdd77-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.872697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.872654 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.872994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.872890 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8scwl\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-kube-api-access-8scwl\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:13.984246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:13.984214 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2"] Nov 01 16:02:14.001758 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.001674 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2"] Nov 01 16:02:14.002013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.001865 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.025714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.025685 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:14.066163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066124 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066179 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869649a0-d0cd-4236-ab36-567fc7890712-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066249 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6wv\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-kube-api-access-mc6wv\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066377 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066436 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066515 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066554 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869649a0-d0cd-4236-ab36-567fc7890712-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066571 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.066753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.066616 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.149339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.149299 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc"] Nov 01 16:02:14.157522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.157470 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.157666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.157576 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.157666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.157623 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.167782 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.167751 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.167918 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.167810 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869649a0-d0cd-4236-ab36-567fc7890712-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.167918 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.167843 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6wv\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-kube-api-access-mc6wv\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.167947 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168016 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168128 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168188 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869649a0-d0cd-4236-ab36-567fc7890712-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168219 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168256 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168703 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168703 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.168866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.168855 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869649a0-d0cd-4236-ab36-567fc7890712-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.169862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.169232 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.170172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.170147 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.170972 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.170948 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.171647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.171602 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869649a0-d0cd-4236-ab36-567fc7890712-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.181323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.181135 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.185073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.184558 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6wv\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-kube-api-access-mc6wv\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.226963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.226932 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8"] Nov 01 16:02:14.232283 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.232262 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.243679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.243655 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8"] Nov 01 16:02:14.269549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269364 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db969f4-c84d-47f7-8240-130d785859c1-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.269549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269495 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.269549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269530 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db969f4-c84d-47f7-8240-130d785859c1-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.269823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269577 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.269823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269651 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.269823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269673 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.269823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269765 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.269823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269809 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvzx\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-kube-api-access-8xvzx\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.270090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.269898 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.324524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.324493 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:14.326117 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:02:14.326089 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97794338_5b3b_447a_9aa3_221d194cdd77.slice/crio-6286b9a32b16f45b4721a19423f9f58a87ae54dcfd40a1744da927442d632fb3 WatchSource:0}: Error finding container 6286b9a32b16f45b4721a19423f9f58a87ae54dcfd40a1744da927442d632fb3: Status 404 returned error can't find the container with id 6286b9a32b16f45b4721a19423f9f58a87ae54dcfd40a1744da927442d632fb3 Nov 01 16:02:14.327312 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.327281 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn"] Nov 01 16:02:14.329469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.329432 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.329575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.329517 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.329631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.329587 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.371506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371432 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.371506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371483 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvzx\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-kube-api-access-8xvzx\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.371734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371571 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.371734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371674 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db969f4-c84d-47f7-8240-130d785859c1-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.371851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371781 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.371851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371816 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db969f4-c84d-47f7-8240-130d785859c1-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.371960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371862 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.371960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371938 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.372079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.371964 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.374090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.372197 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.374090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.372258 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.374090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.373088 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db969f4-c84d-47f7-8240-130d785859c1-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.374090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.373228 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.374090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.373485 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.375384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.375356 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db969f4-c84d-47f7-8240-130d785859c1-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.375953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.375930 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.380577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.380551 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvzx\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-kube-api-access-8xvzx\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.382813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.382763 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.550939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.550854 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:14.667100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.664383 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2"] Nov 01 16:02:14.668476 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.668424 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" event={"ID":"6f3ab62e-a9b7-4485-87f6-aa6f3519d446","Type":"ContainerStarted","Data":"7dbe62465e999ec72d5210b8cefa2ea1f7211af55bbf98517570148abe07946e"} Nov 01 16:02:14.668631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.668486 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" event={"ID":"6f3ab62e-a9b7-4485-87f6-aa6f3519d446","Type":"ContainerStarted","Data":"6ca56b3abdc18ad9d53ebac1106e8e1dfdc665a9788e5adb5487c7f129015846"} Nov 01 16:02:14.671883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.671784 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" event={"ID":"97794338-5b3b-447a-9aa3-221d194cdd77","Type":"ContainerStarted","Data":"ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d"} Nov 01 16:02:14.671883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.671835 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" event={"ID":"97794338-5b3b-447a-9aa3-221d194cdd77","Type":"ContainerStarted","Data":"6286b9a32b16f45b4721a19423f9f58a87ae54dcfd40a1744da927442d632fb3"} Nov 01 16:02:14.686676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.686642 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.687201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.687176 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.687365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.687348 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.719731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.719348 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podStartSLOduration=1.71932197 podStartE2EDuration="1.71932197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:14.70790946 +0000 UTC m=+874.852364114" watchObservedRunningTime="2026-11-01 16:02:14.71932197 +0000 UTC m=+874.863776639" Nov 01 16:02:14.762739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.760700 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podStartSLOduration=1.760680928 podStartE2EDuration="1.760680928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:14.738654791 +0000 UTC m=+874.883109435" watchObservedRunningTime="2026-11-01 16:02:14.760680928 +0000 UTC m=+874.905135569" Nov 01 16:02:14.847752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.847619 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:14.848848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.848810 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8"] Nov 01 16:02:14.903073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.902996 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.903347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.903321 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.903486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:14.903468 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:15.025845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.025808 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:15.027470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.027440 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:15.027601 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.027492 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:15.742152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.742106 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" event={"ID":"5db969f4-c84d-47f7-8240-130d785859c1","Type":"ContainerStarted","Data":"66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104"} Nov 01 16:02:15.746191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.746154 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" event={"ID":"5db969f4-c84d-47f7-8240-130d785859c1","Type":"ContainerStarted","Data":"1b9c619b699288a4ef80531cae5dd5c25761e60432076db9984ed91de7be3bff"} Nov 01 16:02:15.768061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.754078 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" event={"ID":"869649a0-d0cd-4236-ab36-567fc7890712","Type":"ContainerStarted","Data":"3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2"} Nov 01 16:02:15.768061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.754138 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" event={"ID":"869649a0-d0cd-4236-ab36-567fc7890712","Type":"ContainerStarted","Data":"432840ce639693f6d2068ecb72d986fea03e9c35a533bdfa027071edee3a461b"} Nov 01 16:02:15.793555 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.793485 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podStartSLOduration=1.7934624270000001 podStartE2EDuration="1.793462427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:15.774699708 +0000 UTC m=+875.919154351" watchObservedRunningTime="2026-11-01 16:02:15.793462427 +0000 UTC m=+875.937917072" Nov 01 16:02:15.886708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.869175 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:15.886708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.869270 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:15.908068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:15.887244 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podStartSLOduration=2.887220886 podStartE2EDuration="2.887220886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:15.801739803 +0000 UTC m=+875.946194445" watchObservedRunningTime="2026-11-01 16:02:15.887220886 +0000 UTC m=+876.031675527" Nov 01 16:02:16.037504 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.031668 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.037504 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.031734 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:16.332647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.332531 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.332904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.332860 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:16.333415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.333331 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:16.555974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.555940 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:16.556330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.556304 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.556527 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.556499 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:16.849067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.848711 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.849067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:16.848775 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:17.026406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.026369 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.026815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.026679 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:17.143448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.143323 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:02:17.203255 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.203190 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.226265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.226201 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.326416 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.326379 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.326702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.326671 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:17.468122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.464850 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.492955 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.492892 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.544189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.544132 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.552517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.552481 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.552876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.552827 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:17.570154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.570101 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.730122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.728173 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:17.859180 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.859129 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Nov 01 16:02:17.940067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.892867 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.940067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:17.892928 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:18.029451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.029413 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.029738 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.029702 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:18.242070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.240701 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.320279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.300995 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:02:18.385958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.315152 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.489927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.351459 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.490259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.490229 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:18.490402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.482465 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.491897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.491872 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:02:18.503159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.502176 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.529228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.529191 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:02:18.545691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.545652 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.547227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.545886 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.592124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.588269 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.625064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.601229 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:02:18.625064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.588510 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:18.655064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.634733 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.662260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.662211 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:02:18.676681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.676642 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.794117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.781626 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:18.794117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.787692 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:02:18.974360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:18.974259 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.007062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.004842 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.007062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.004909 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:19.007062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.005494 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:02:19.034073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.028772 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 01 16:02:19.090255 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.090209 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.090535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.090506 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:19.229127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.215603 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:02:19.229127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.216982 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.240074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.232102 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.275491 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.241138 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.352224 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.352189 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.367251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.367206 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:19.492199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.492150 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.511881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.511838 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:02:19.539216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.539173 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:02:19.555212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.555165 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.582409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.555575 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.582671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.582631 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:19.587131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.555637 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.615066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.591293 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:02:19.649174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.649121 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.680409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.680366 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:19.693066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.681956 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:02:19.903342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.887700 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:02:19.948066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.889138 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.074065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.989295 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.074065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:19.989356 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:20.074065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.030126 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:02:20.100065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.087752 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.100065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.087819 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:20.100065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.087926 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.131070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.100675 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.274063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.273301 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.274063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.273650 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:02:20.298065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.276620 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.298065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.276854 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:02:20.407073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.406984 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.407254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.407090 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:20.416073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.414884 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:02:20.416073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.415006 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:02:20.416073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.415090 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:02:20.416073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.415168 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:02:20.416073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.415237 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:02:20.496932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.496881 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.566117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.565155 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.566117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.565553 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:02:20.595421 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.567415 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.595421 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.567462 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:20.633385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.605253 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.633385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.611113 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.633385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.629558 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:02:20.633385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.630799 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.649983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.649943 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:20.707270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.686386 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:02:20.905436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.851867 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.905436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.851928 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:20.906629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:20.906577 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.062076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.050521 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.148065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.119208 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.162164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.119830 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:02:21.210106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.146332 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:02:21.235073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.162140 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:21.377066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.373085 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.377066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.373148 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:21.427349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.427240 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.427937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.427903 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:02:21.428408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.428383 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:02:21.520064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.514257 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.585208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.585155 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.588412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.588093 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.588637 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.588610 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:21.612184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.612142 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.647462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.647419 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.750125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.659463 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.798002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.769841 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:02:21.831203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.831112 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.874389 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.853654 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:21.874389 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.853711 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:21.921388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.921340 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:21.955882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:21.953708 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:02:22.375661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.375601 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:02:22.405106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.381694 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.447522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.447405 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.447858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.447802 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:22.486085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.479480 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 01 16:02:22.547434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.520735 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.547434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.520800 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:22.547434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.520902 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.547434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.532572 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:02:22.563089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.560592 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:22.563089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.560670 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:22.598249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.598192 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.616480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.615839 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:02:22.616480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.616202 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:02:22.655750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.652009 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.718114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.692652 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:22.778641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.777243 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:02:22.884072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.884011 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:02:22.907178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.894518 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:02:22.938088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:22.927994 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:02:23.017549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.017519 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:02:23.028958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.028932 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:02:23.036826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.036791 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:23.038347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.037318 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:02:23.038583 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.038561 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.046057 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.038757 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 01 16:02:23.051064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.047721 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.051064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.047806 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:23.051470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.051450 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:02:23.141616 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.141578 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:02:23.141806 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.141671 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:02:23.326306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.325938 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.326306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.325997 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 01 16:02:23.551828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.551283 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.551828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.551343 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 01 16:02:23.848893 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.848817 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 01 16:02:23.848893 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:23.848871 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 01 16:02:24.034062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.034016 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:24.034959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.034936 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:24.036435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.036417 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:02:24.461088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.457332 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:24.461088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.458142 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:24.461088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.459793 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:02:24.854560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.854525 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:24.855432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.855410 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:24.856109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:24.856089 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:02:25.054256 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:25.054227 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:25.054752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:25.054733 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:25.055887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:25.055860 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:02:31.322831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.297245 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.396594 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.396550 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.450191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.450138 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.651259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.651149 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.717065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.690882 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.745427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.718100 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.746097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.718146 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.762173 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.762133 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.888184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.888126 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.927242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:31.927138 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.203881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.203769 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.336063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.334252 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.336063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.334864 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:02:32.370925 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.370891 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:02:32.458285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.458178 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.465329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.465288 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:02:32.506589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.506524 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.508501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.508477 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:02:32.690053 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.689863 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.740275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.740235 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.741489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.741462 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:02:32.784069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.770096 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:02:32.880235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.790137 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:02:32.880235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.773144 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.880235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:32.793164 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:02:33.126909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:33.126814 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:02:33.157624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:33.157589 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:02:33.338920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:33.338886 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:02:40.204526 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:40.204490 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:02:40.205003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:40.204494 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:02:40.867376 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:02:40.867342 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:02:43.840739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.840701 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4"] Nov 01 16:02:43.859307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.859273 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.872547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.872511 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4"] Nov 01 16:02:43.989317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989279 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989336 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989466 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989632 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989610 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b15167e-6477-40de-8a0e-f072897a52ae-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989656 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b15167e-6477-40de-8a0e-f072897a52ae-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989738 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989799 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989776 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989805 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:43.989842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:43.989824 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sljv\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-kube-api-access-8sljv\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091220 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091319 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091443 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091589 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b15167e-6477-40de-8a0e-f072897a52ae-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091644 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b15167e-6477-40de-8a0e-f072897a52ae-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091765 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091781 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.091866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091797 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.092015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091867 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.092015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091906 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sljv\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-kube-api-access-8sljv\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.092015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.091926 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.092258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.092179 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.092895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.092856 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.093235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.093208 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b15167e-6477-40de-8a0e-f072897a52ae-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.094931 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.094904 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.095480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.095416 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b15167e-6477-40de-8a0e-f072897a52ae-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.099373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.099350 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.100632 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.100611 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sljv\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-kube-api-access-8sljv\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.187347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.187305 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:44.484804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.484770 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4"] Nov 01 16:02:44.487768 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:02:44.487742 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b15167e_6477_40de_8a0e_f072897a52ae.slice/crio-58153db2f4e80769d03950c635afd82a5116134906ed757c1fbeae36ba047b6d WatchSource:0}: Error finding container 58153db2f4e80769d03950c635afd82a5116134906ed757c1fbeae36ba047b6d: Status 404 returned error can't find the container with id 58153db2f4e80769d03950c635afd82a5116134906ed757c1fbeae36ba047b6d Nov 01 16:02:44.490992 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.490956 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.491145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.491026 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.491145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:44.491140 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:45.489115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:45.489075 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" event={"ID":"3b15167e-6477-40de-8a0e-f072897a52ae","Type":"ContainerStarted","Data":"15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb"} Nov 01 16:02:45.489526 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:45.489139 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" event={"ID":"3b15167e-6477-40de-8a0e-f072897a52ae","Type":"ContainerStarted","Data":"58153db2f4e80769d03950c635afd82a5116134906ed757c1fbeae36ba047b6d"} Nov 01 16:02:45.512541 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:45.512492 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podStartSLOduration=2.512477863 podStartE2EDuration="2.512477863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:45.509834457 +0000 UTC m=+905.654289097" watchObservedRunningTime="2026-11-01 16:02:45.512477863 +0000 UTC m=+905.656932503" Nov 01 16:02:46.188444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:46.188403 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:46.190127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:46.190099 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 16:02:46.190261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:46.190149 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 16:02:47.204068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:47.199270 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 16:02:47.204068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:47.199332 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 16:02:48.188659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:48.188624 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 16:02:48.188845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:48.188682 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 16:02:49.193064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:49.191600 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 16:02:49.193064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:49.191690 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 16:02:50.188286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:50.188252 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 16:02:50.188450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:50.188309 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 16:02:51.189060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:51.188487 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 01 16:02:51.189060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:51.188570 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 01 16:02:52.193221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:52.193186 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:52.193638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:52.193616 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:02:52.194379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:02:52.194358 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:03:01.172066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:01.163187 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.208196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:01.208137 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.380272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:01.380221 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.166174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.166122 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:03:02.220028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.209743 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.254780 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.254731 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:03:02.274076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.273946 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:03:02.359106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.355295 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.397578 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.380146 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:03:02.397884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.380868 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.399188 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.398473 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:03:02.486575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.486533 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.497983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.497936 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.587504 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.517488 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.942354 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:02.942225 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:03:03.151564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.151497 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.171872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.171830 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:03:03.297319 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.297255 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:03.297923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.297885 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.397923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.397874 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.627903 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.398240 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.627903 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.398327 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.627903 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.398404 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.627903 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.562268 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:03.627903 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.579890 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.627903 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.581544 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.652088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.628930 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:03:03.652088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.629225 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:03:03.652088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.629319 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:03:03.652840 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.652791 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:03.850676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.850624 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.852135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.852092 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:03.879126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.858913 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.959491 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:03.959444 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:03:04.141140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.117095 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.189066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.172461 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.198087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.191954 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.317063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.315572 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:04.476126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.475994 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:03:04.476126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.476097 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:03:04.663066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.621583 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.663775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.648619 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:03:04.665734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.648656 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:03:04.706287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.689738 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:03:04.706287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.689812 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:03:04.706287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.648693 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:04.756117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.648729 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.756117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.648676 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 01 16:03:04.756117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.706212 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:04.756117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.740362 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:03:04.756117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.740424 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:03:04.756117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.706248 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 01 16:03:04.862651 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.862603 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:04.863397 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.863370 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:03:04.948076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.945906 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.948589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.948547 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:03:04.961226 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.961126 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.998066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:04.987344 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:03:05.125607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.125476 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.282272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.279355 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:03:05.282584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.282533 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:03:05.383155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.350920 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:05.384603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.376639 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Nov 01 16:03:05.384887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.384864 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:03:05.668452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.668156 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.700105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.668733 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.700370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.668759 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.768192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.768143 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.769853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.768405 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.776000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.768584 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.780160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.768628 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:05.965346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.939994 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:06.007986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.945223 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.007986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.946316 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:03:06.008395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:05.993342 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.186064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.185263 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.278063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.277856 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.289074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.288579 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:03:06.321795 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.321750 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.388258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.387138 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.608122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.605540 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:03:06.740469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.726071 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.831062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.791127 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:06.831062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.791598 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:03:06.831062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.829146 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:06.924148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.905360 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:03:06.948189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.924174 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:03:06.948189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:06.928338 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:03:07.092083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.092015 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 01 16:03:07.093910 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.093864 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.128476 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.092016 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.151238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.151210 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:03:07.158159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.158128 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:03:07.198418 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.198333 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:03:07.200933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.200900 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.201685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.201664 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:03:07.205056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.205024 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:03:07.205252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.205237 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:03:07.205381 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.205369 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:03:07.205489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.205479 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:03:07.216077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.207364 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:03:07.216077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.214427 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:03:07.231900 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:07.231867 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:03:10.878422 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:03:10.878354 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:03:28.566711 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.566614 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps"] Nov 01 16:03:28.576340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.576302 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.587797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.587767 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps"] Nov 01 16:03:28.630851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.630807 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631319 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbdw\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-kube-api-access-8dbdw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631455 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631518 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631553 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631646 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631633 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631669 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631794 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.631859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.631842 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.733690 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.733658 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.733862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.733755 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.733862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.733784 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.733862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.733823 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.733993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.733910 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbdw\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-kube-api-access-8dbdw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.733993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.733976 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734016 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734060 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734165 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734217 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734422 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734515 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734709 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.734926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.734906 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.736372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.736344 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.736563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.736533 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.742800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.742769 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbdw\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-kube-api-access-8dbdw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.744347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.744324 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:28.896494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:28.896402 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:29.200783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:29.200751 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps"] Nov 01 16:03:29.202914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:29.202879 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:29.203026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:29.202963 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:29.203026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:29.203012 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:30.115114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.115077 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" event={"ID":"27ed155f-4b1d-4880-a77c-3dcc0f38466c","Type":"ContainerStarted","Data":"6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f"} Nov 01 16:03:30.115592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.115126 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" event={"ID":"27ed155f-4b1d-4880-a77c-3dcc0f38466c","Type":"ContainerStarted","Data":"e0fb35eda5c88d1c33610bade17bf08e0e71b81c3978e1123322315aceb0e893"} Nov 01 16:03:30.138048 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.137976 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podStartSLOduration=2.137957887 podStartE2EDuration="2.137957887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:30.134357334 +0000 UTC m=+950.278811987" watchObservedRunningTime="2026-11-01 16:03:30.137957887 +0000 UTC m=+950.282412528" Nov 01 16:03:30.321260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.321222 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6948-350f21"] Nov 01 16:03:30.327132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.327097 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:30.328561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.328531 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-350f21"] Nov 01 16:03:30.331330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.331306 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwjk\"" Nov 01 16:03:30.482158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.482074 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgj9w\" (UniqueName: \"kubernetes.io/projected/d2d19307-948e-4b8d-aced-d9708824ca24-kube-api-access-tgj9w\") pod \"test-curl-6948-350f21\" (UID: \"d2d19307-948e-4b8d-aced-d9708824ca24\") " pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:30.584192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.584152 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgj9w\" (UniqueName: \"kubernetes.io/projected/d2d19307-948e-4b8d-aced-d9708824ca24-kube-api-access-tgj9w\") pod \"test-curl-6948-350f21\" (UID: \"d2d19307-948e-4b8d-aced-d9708824ca24\") " pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:30.592427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.592393 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgj9w\" (UniqueName: \"kubernetes.io/projected/d2d19307-948e-4b8d-aced-d9708824ca24-kube-api-access-tgj9w\") pod \"test-curl-6948-350f21\" (UID: \"d2d19307-948e-4b8d-aced-d9708824ca24\") " pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:30.644461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.644418 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:30.897160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.897121 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:30.897839 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.897812 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:30.897946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.897865 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:30.976965 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:03:30.976924 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2d19307_948e_4b8d_aced_d9708824ca24.slice/crio-d89d99fac7ad5424dd41222803ea4e580b2acdf3c81ebffdd62c4efffc82b150 WatchSource:0}: Error finding container d89d99fac7ad5424dd41222803ea4e580b2acdf3c81ebffdd62c4efffc82b150: Status 404 returned error can't find the container with id d89d99fac7ad5424dd41222803ea4e580b2acdf3c81ebffdd62c4efffc82b150 Nov 01 16:03:30.978391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.978361 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-350f21"] Nov 01 16:03:30.979373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:30.979354 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:03:31.131592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:31.131488 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-350f21" event={"ID":"d2d19307-948e-4b8d-aced-d9708824ca24","Type":"ContainerStarted","Data":"d89d99fac7ad5424dd41222803ea4e580b2acdf3c81ebffdd62c4efffc82b150"} Nov 01 16:03:31.897475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:31.897433 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:31.897688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:31.897529 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:32.896904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:32.896866 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:32.897349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:32.896918 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:33.916077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:33.912176 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:33.916077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:33.912251 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:34.681065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:34.680141 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:34.798201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:34.798146 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:34.826201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:34.826130 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.065065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:34.957185 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.065065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:34.976563 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.065065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.001084 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:35.065065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.054463 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:35.065065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.019390 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:03:35.109851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.109146 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.232062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.231589 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.237068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.236799 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.245115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.241542 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.245115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.242256 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.331118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.330159 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.391065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.390142 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.692118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.681003 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.692118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.681762 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:03:35.718065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.700494 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.718065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.704672 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.778065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.772349 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Nov 01 16:03:35.821063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.816508 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:35.821063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.817177 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:03:35.868064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.867307 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.868064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:35.867914 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:03:36.013066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.010175 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:36.013066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.010262 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:36.079592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.079538 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.119268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.117108 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.119268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.117905 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:03:36.119268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.118219 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:03:36.227066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.213394 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.309562 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.309458 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.357073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.314889 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:36.357073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.314925 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:03:36.357073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.314947 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.357073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.349563 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405548 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405635 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405691 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405729 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405781 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405814 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405862 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405897 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:03:36.419080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.405949 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:03:36.719128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.717802 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.719128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.718452 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.719346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.719128 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:03:36.890064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:36.885095 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:37.049567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.049527 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:37.049843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.049814 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:37.101530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.101477 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.155588 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.155537 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.183067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.179833 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.243122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.182849 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:03:37.245433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.205002 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.428307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.428181 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.437626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.437570 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.463251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.463202 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.547063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.512382 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:37.559064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.555124 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:37.805337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.800460 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.125148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.876348 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.127655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.878283 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.127879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.912920 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:38.128236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.926615 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:38.128426 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.128398 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:38.128537 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.058589 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.128652 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.128631 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.128761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:37.936857 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.188632 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.176280 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.191177 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.191136 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 01 16:03:38.192627 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.192599 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:03:38.237618 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.237573 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.237901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.237868 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.245031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.244985 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.266068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.257145 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.266068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.257462 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.266068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.257508 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.266068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.261315 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.572064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.571155 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.580111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.575226 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:38.596112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.591883 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:03:38.596112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.593087 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:03:38.601542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.601501 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:03:38.767065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.761798 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.767065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.761892 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.900589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.900471 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:38.900883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.900832 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:38.933825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.933780 4911 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.951194 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.951142 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.976065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:38.963075 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.131077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.125274 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:03:39.160825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.160692 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.163679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.161149 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:39.164073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.161399 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:03:39.170634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.170607 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:03:39.218066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.192918 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.219263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.191293 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.249373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.246162 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.270489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.270441 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.325082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.271055 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.635372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.635233 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:39.637067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.636811 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:03:39.637219 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.637132 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:39.649067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.648979 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.653767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.649796 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.775618 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.775584 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Nov 01 16:03:39.776450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.775521 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.907329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.907228 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:39.907625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:39.907575 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:40.020510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.020462 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.114071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.094136 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:40.114071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.094203 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.156602 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.155272 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.170126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.170016 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:03:40.250472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.192642 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.251159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.192674 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.251532 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.205891 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:40.251720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.251689 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.251835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.228679 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.252198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.228128 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.252540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.250078 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:40.252682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.252657 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.252789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.246940 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:03:40.278069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.272173 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.278069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.272606 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.539249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.539208 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:40.539567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.539515 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.641066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.636788 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:40.641066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.637979 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:40.641066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.638290 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:03:40.665331 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.660264 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.665331 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.660688 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.903160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.901105 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:40.903160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.901165 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:41.011067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.993299 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:41.011067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:40.993363 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.122380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.122335 4911 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:41.122668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.122632 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.209419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.209305 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.254105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.253123 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.254105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.253506 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:03:41.254105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.253772 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:41.254105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.254015 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:03:41.254493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.254309 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:41.254551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.254518 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:03:41.278610 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.278566 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.294512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.294467 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:03:41.625406 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:03:41.625366 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:03:41.651835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.651789 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.652770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.652734 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.660070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.654596 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:41.667082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.660693 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.667082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.661524 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.912876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.912779 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:41.925070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:41.921363 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:42.143428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.143354 4911 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:42.143732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.143678 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.217286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.216127 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.263418 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.264160 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.264573 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.264904 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.265221 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.265560 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.309566 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.323196 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.351789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.350350 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.520129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.508870 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": context deadline exceeded" Nov 01 16:03:42.662196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.662147 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:03:42.671070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.662743 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:03:42.679238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.666487 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.687075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.666533 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:03:42.746072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.666555 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:03:42.751891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.691768 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 01 16:03:42.908965 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.898081 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:42.911460 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:42.911416 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:43.217399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.216373 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.270685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.270642 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.293061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.280098 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:03:43.293061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.280385 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:03:43.293061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.280580 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:03:43.312155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.300096 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:43.323990 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.323949 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.324469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.312407 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.335085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.300568 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:03:43.501112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.490526 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:03:43.501112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.492311 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:03:43.620902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.620871 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:03:43.623425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.623404 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:03:43.699704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.699654 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:03:43.717259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.716811 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:03:43.790384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.780949 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:03:43.790384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.781011 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:03:43.790384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.776701 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.800131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.798313 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:03:43.800131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.799656 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:03:43.803322 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.802118 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:03:43.807759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.806533 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:03:43.807759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.806577 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:03:43.810205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.810185 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:03:43.840974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.840944 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:03:43.898648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.898614 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:43.899108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.899059 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 16:03:43.899257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:43.899127 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 16:03:44.832007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:44.831936 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-350f21" event={"ID":"d2d19307-948e-4b8d-aced-d9708824ca24","Type":"ContainerStarted","Data":"3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841"} Nov 01 16:03:44.849057 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:44.848987 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6948-350f21" podStartSLOduration=1.343012446 podStartE2EDuration="14.848972872s" totalImagesPullingTime="13.505960426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:44.844723427 +0000 UTC m=+964.989178070" watchObservedRunningTime="2026-11-01 16:03:44.848972872 +0000 UTC m=+964.993427512" Nov 01 16:03:45.526289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.522803 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6948-350f21"] Nov 01 16:03:45.531403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.528325 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6948-350f21"] Nov 01 16:03:45.679307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.679272 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6948-df4c2d"] Nov 01 16:03:45.680574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.680544 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d2d19307-948e-4b8d-aced-d9708824ca24" containerName="test-curl-6948-350f21" Nov 01 16:03:45.680574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.680573 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d19307-948e-4b8d-aced-d9708824ca24" containerName="test-curl-6948-350f21" Nov 01 16:03:45.680851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.680830 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d19307-948e-4b8d-aced-d9708824ca24" containerName="test-curl-6948-350f21" Nov 01 16:03:45.687238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.687211 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:45.688720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.688692 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-df4c2d"] Nov 01 16:03:45.786540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.786426 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x2rk\" (UniqueName: \"kubernetes.io/projected/33116a18-f6c1-4a67-98fa-ce336852def0-kube-api-access-7x2rk\") pod \"test-curl-6948-df4c2d\" (UID: \"33116a18-f6c1-4a67-98fa-ce336852def0\") " pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:45.888033 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.887991 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x2rk\" (UniqueName: \"kubernetes.io/projected/33116a18-f6c1-4a67-98fa-ce336852def0-kube-api-access-7x2rk\") pod \"test-curl-6948-df4c2d\" (UID: \"33116a18-f6c1-4a67-98fa-ce336852def0\") " pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:45.900587 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.900550 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x2rk\" (UniqueName: \"kubernetes.io/projected/33116a18-f6c1-4a67-98fa-ce336852def0-kube-api-access-7x2rk\") pod \"test-curl-6948-df4c2d\" (UID: \"33116a18-f6c1-4a67-98fa-ce336852def0\") " pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:45.939216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.939180 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:45.939381 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:45.939242 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:46.004606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.004571 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:46.312417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.312371 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6948-df4c2d"] Nov 01 16:03:46.414334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.414307 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:46.416483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.416458 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:03:46.851186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.851150 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-df4c2d" event={"ID":"33116a18-f6c1-4a67-98fa-ce336852def0","Type":"ContainerStarted","Data":"7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db"} Nov 01 16:03:46.851186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.851192 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6948-df4c2d" event={"ID":"33116a18-f6c1-4a67-98fa-ce336852def0","Type":"ContainerStarted","Data":"4735fc4ff193ccca38b654e5c398d0e4d77f472a7ddc1afdf7cf0919cd252f4b"} Nov 01 16:03:46.851436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.851242 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6948-350f21" podUID="d2d19307-948e-4b8d-aced-d9708824ca24" containerName="test-curl-6948-350f21" containerID="cri-o://3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841" gracePeriod=2 Nov 01 16:03:46.911119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:46.911026 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6948-df4c2d" podStartSLOduration=1.911006606 podStartE2EDuration="1.911006606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:46.906022304 +0000 UTC m=+967.050476946" watchObservedRunningTime="2026-11-01 16:03:46.911006606 +0000 UTC m=+967.055461249" Nov 01 16:03:47.644529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:47.637288 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6948-df4c2d"] Nov 01 16:03:47.649759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:47.648355 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6948-df4c2d"] Nov 01 16:03:48.871463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:48.870522 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6948-df4c2d" podUID="33116a18-f6c1-4a67-98fa-ce336852def0" containerName="test-curl-6948-df4c2d" containerID="cri-o://7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db" gracePeriod=2 Nov 01 16:03:49.085032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.085005 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:49.137833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.137753 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp"] Nov 01 16:03:49.138840 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.138812 4911 cpu_manager.go:414] "containerMap: removing container" podUID="33116a18-f6c1-4a67-98fa-ce336852def0" containerName="test-curl-6948-df4c2d" Nov 01 16:03:49.138840 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.138839 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33116a18-f6c1-4a67-98fa-ce336852def0" containerName="test-curl-6948-df4c2d" Nov 01 16:03:49.139088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.139071 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="33116a18-f6c1-4a67-98fa-ce336852def0" containerName="test-curl-6948-df4c2d" Nov 01 16:03:49.147082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.144698 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.152749 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.152722 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3664624a-gw-openshift-default-dockercfg-52sv6\"" Nov 01 16:03:49.160731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.158717 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d19307-948e-4b8d-aced-d9708824ca24-kube-api-access-tgj9w\" (UniqueName: \"kubernetes.io/projected/d2d19307-948e-4b8d-aced-d9708824ca24-kube-api-access-tgj9w\") pod \"d2d19307-948e-4b8d-aced-d9708824ca24\" (UID: \"d2d19307-948e-4b8d-aced-d9708824ca24\") " Nov 01 16:03:49.172514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.172489 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp"] Nov 01 16:03:49.186911 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.185801 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d19307-948e-4b8d-aced-d9708824ca24-kube-api-access-tgj9w" pod "d2d19307-948e-4b8d-aced-d9708824ca24" (UID: "d2d19307-948e-4b8d-aced-d9708824ca24"). InnerVolumeSpecName "kube-api-access-tgj9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:49.264833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.264785 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-data\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.264992 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-podinfo\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265135 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wlt\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-kube-api-access-n9wlt\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265251 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-token\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265297 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-socket\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265345 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-credential-socket\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265435 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-certs\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265463 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-envoy\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265649 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istiod-ca-cert\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.265862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.265816 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgj9w\" (UniqueName: \"kubernetes.io/projected/d2d19307-948e-4b8d-aced-d9708824ca24-kube-api-access-tgj9w\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:49.369065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.369004 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-data\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.369440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.369416 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-podinfo\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.369619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.369603 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wlt\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-kube-api-access-n9wlt\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.369778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.369764 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-token\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.369950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.369935 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-socket\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.370105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.370088 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-credential-socket\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.370232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.370214 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-certs\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.371599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.371578 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-envoy\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.371873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.371855 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istiod-ca-cert\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.412210 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.372899 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istiod-ca-cert\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.412488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.382885 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-socket\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.412605 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.383228 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-credential-socket\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.412689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.383594 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-certs\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.412770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.371131 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-data\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.412861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.388630 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-podinfo\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.412943 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.390872 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-envoy\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.415484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.415452 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wlt\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-kube-api-access-n9wlt\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.448068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.445822 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-token\") pod \"e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.497624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.497337 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:49.815612 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:03:49.815582 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30afe6eb_2bc1_42a8_b21a_ab067ac95831.slice/crio-25471b8e8e143f17da6b0225f4b9793c26c0bf652a2021d22c85141035bd38cb WatchSource:0}: Error finding container 25471b8e8e143f17da6b0225f4b9793c26c0bf652a2021d22c85141035bd38cb: Status 404 returned error can't find the container with id 25471b8e8e143f17da6b0225f4b9793c26c0bf652a2021d22c85141035bd38cb Nov 01 16:03:49.816230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.816207 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp"] Nov 01 16:03:49.818550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.818512 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:49.818634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.818598 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:49.818675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.818640 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:49.881167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.881135 4911 generic.go:356] "Generic (PLEG): container finished" podID="d2d19307-948e-4b8d-aced-d9708824ca24" containerID="3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841" exitCode=137 Nov 01 16:03:49.881673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.881540 4911 scope.go:122] "RemoveContainer" containerID="3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841" Nov 01 16:03:49.881789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.881555 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:49.884348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.883990 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" event={"ID":"30afe6eb-2bc1-42a8-b21a-ab067ac95831","Type":"ContainerStarted","Data":"25471b8e8e143f17da6b0225f4b9793c26c0bf652a2021d22c85141035bd38cb"} Nov 01 16:03:49.903665 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:03:49.903636 4911 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-350f21\" is forbidden: User \"system:node:ip-10-0-143-101.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-101.us-east-2.compute.internal' and this object" podUID="d2d19307-948e-4b8d-aced-d9708824ca24" pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:49.906226 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.906198 4911 scope.go:122] "RemoveContainer" containerID="3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841" Nov 01 16:03:49.906628 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:03:49.906601 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841\": container with ID starting with 3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841 not found: ID does not exist" containerID="3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841" Nov 01 16:03:49.906732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.906642 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841"} err="failed to get container status \"3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841\": rpc error: code = NotFound desc = could not find container \"3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841\": container with ID starting with 3245a4d4be730a0d6ad22b548121fc9f67f1d25a7321e5c5bdd4f4f83cae5841 not found: ID does not exist" Nov 01 16:03:49.970443 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:03:49.970410 4911 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-350f21\" is forbidden: User \"system:node:ip-10-0-143-101.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-101.us-east-2.compute.internal' and this object" podUID="d2d19307-948e-4b8d-aced-d9708824ca24" pod="opendatahub/test-curl-6948-350f21" Nov 01 16:03:49.982026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:49.981986 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d19307-948e-4b8d-aced-d9708824ca24" path="/var/lib/kubelet/pods/d2d19307-948e-4b8d-aced-d9708824ca24/volumes" Nov 01 16:03:50.896669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:50.896630 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" event={"ID":"30afe6eb-2bc1-42a8-b21a-ab067ac95831","Type":"ContainerStarted","Data":"748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69"} Nov 01 16:03:50.920272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:50.920226 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" podStartSLOduration=1.92020806 podStartE2EDuration="1.92020806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:50.918343413 +0000 UTC m=+971.062798056" watchObservedRunningTime="2026-11-01 16:03:50.92020806 +0000 UTC m=+971.064662702" Nov 01 16:03:51.145815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.145725 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:51.308898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.308865 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33116a18-f6c1-4a67-98fa-ce336852def0-kube-api-access-7x2rk\" (UniqueName: \"kubernetes.io/projected/33116a18-f6c1-4a67-98fa-ce336852def0-kube-api-access-7x2rk\") pod \"33116a18-f6c1-4a67-98fa-ce336852def0\" (UID: \"33116a18-f6c1-4a67-98fa-ce336852def0\") " Nov 01 16:03:51.315692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.315656 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33116a18-f6c1-4a67-98fa-ce336852def0-kube-api-access-7x2rk" pod "33116a18-f6c1-4a67-98fa-ce336852def0" (UID: "33116a18-f6c1-4a67-98fa-ce336852def0"). InnerVolumeSpecName "kube-api-access-7x2rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:51.410939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.410858 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x2rk\" (UniqueName: \"kubernetes.io/projected/33116a18-f6c1-4a67-98fa-ce336852def0-kube-api-access-7x2rk\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:51.500281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.498736 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:51.509141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.509107 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:51.906827 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.906791 4911 generic.go:356] "Generic (PLEG): container finished" podID="33116a18-f6c1-4a67-98fa-ce336852def0" containerID="7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db" exitCode=137 Nov 01 16:03:51.907377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.906907 4911 scope.go:122] "RemoveContainer" containerID="7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db" Nov 01 16:03:51.907377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.906951 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:51.907377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.907333 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:51.911202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.911180 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:03:51.919304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.919177 4911 scope.go:122] "RemoveContainer" containerID="7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db" Nov 01 16:03:51.919469 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:03:51.919452 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db\": container with ID starting with 7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db not found: ID does not exist" containerID="7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db" Nov 01 16:03:51.919512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.919477 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db"} err="failed to get container status \"7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db\": rpc error: code = NotFound desc = could not find container \"7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db\": container with ID starting with 7de11c3fdb60ccaeccf9136dedb9ea1bba4aeef87e9365a4eecd4d31587972db not found: ID does not exist" Nov 01 16:03:51.935112 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:03:51.935074 4911 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6948-df4c2d\" is forbidden: User \"system:node:ip-10-0-143-101.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-101.us-east-2.compute.internal' and this object" podUID="33116a18-f6c1-4a67-98fa-ce336852def0" pod="opendatahub/test-curl-6948-df4c2d" Nov 01 16:03:51.996684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:51.996640 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33116a18-f6c1-4a67-98fa-ce336852def0" path="/var/lib/kubelet/pods/33116a18-f6c1-4a67-98fa-ce336852def0/volumes" Nov 01 16:03:57.276246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.274687 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4"] Nov 01 16:03:57.280934 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.280225 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.287969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.287943 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-47e1515a-openshift-default-dockercfg-9xn2d\"" Nov 01 16:03:57.295053 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.294996 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4"] Nov 01 16:03:57.411067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.406079 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-data\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.411067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.406170 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-token\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.411067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.406321 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6lf\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-kube-api-access-6j6lf\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.411067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.406370 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-socket\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.411067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.406411 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istiod-ca-cert\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.411067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.406457 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-envoy\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.444440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.444404 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-credential-socket\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.461094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.456162 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-podinfo\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.461094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.456284 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-certs\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.594192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.594092 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-certs\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.615463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.615430 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-data\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.616129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.616105 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-token\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.616754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.616731 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6lf\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-kube-api-access-6j6lf\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.601397 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-certs\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.615998 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-data\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.641314 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-socket\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.635429 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-socket\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.645162 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istiod-ca-cert\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.645218 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-envoy\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.645340 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-credential-socket\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.649075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.645385 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-podinfo\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.663703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.663669 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6lf\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-kube-api-access-6j6lf\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.664455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.664428 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-credential-socket\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.664832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.664813 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-token\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.666250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.666190 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istiod-ca-cert\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.698601 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.697390 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-envoy\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.713646 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.713612 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-podinfo\") pod \"e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:57.924702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:57.924614 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:03:58.249933 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:03:58.249897 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ffd17_2932_46d4_96fa_8b083ba5ebdc.slice/crio-8b298c7170b96640ac65b4774bc12b9c8aa8066cd3d06d6711525275fa5ee90f WatchSource:0}: Error finding container 8b298c7170b96640ac65b4774bc12b9c8aa8066cd3d06d6711525275fa5ee90f: Status 404 returned error can't find the container with id 8b298c7170b96640ac65b4774bc12b9c8aa8066cd3d06d6711525275fa5ee90f Nov 01 16:03:58.249933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:58.249911 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4"] Nov 01 16:03:58.253427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:58.253393 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:58.253535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:58.253466 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:58.253535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:58.253502 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:59.013102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:59.013058 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" event={"ID":"e92ffd17-2932-46d4-96fa-8b083ba5ebdc","Type":"ContainerStarted","Data":"8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09"} Nov 01 16:03:59.013522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:59.013136 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" event={"ID":"e92ffd17-2932-46d4-96fa-8b083ba5ebdc","Type":"ContainerStarted","Data":"8b298c7170b96640ac65b4774bc12b9c8aa8066cd3d06d6711525275fa5ee90f"} Nov 01 16:03:59.037724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:59.037664 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podStartSLOduration=2.037650818 podStartE2EDuration="2.037650818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:59.032929025 +0000 UTC m=+979.177383687" watchObservedRunningTime="2026-11-01 16:03:59.037650818 +0000 UTC m=+979.182105458" Nov 01 16:03:59.961942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:03:59.953142 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:04:00.163573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:00.163527 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:04:01.041079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.040961 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:04:01.042309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.042280 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:04:01.660251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.660210 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth"] Nov 01 16:04:01.670564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.670518 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.675390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.675354 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth"] Nov 01 16:04:01.679480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.679456 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3664624a-serving-cert\"" Nov 01 16:04:01.732486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.732445 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmxv\" (UniqueName: \"kubernetes.io/projected/9eecfa99-af26-4a92-9de5-dbf5afdb1609-kube-api-access-xrmxv\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.732681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.732570 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eecfa99-af26-4a92-9de5-dbf5afdb1609-openshift-service-ca\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.732681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.732619 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-maas-api-tls\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.732769 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.732681 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-metrics-certs\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.834243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.834202 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-metrics-certs\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.848272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.839095 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-metrics-certs\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.848886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.848858 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmxv\" (UniqueName: \"kubernetes.io/projected/9eecfa99-af26-4a92-9de5-dbf5afdb1609-kube-api-access-xrmxv\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.849099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.849078 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eecfa99-af26-4a92-9de5-dbf5afdb1609-openshift-service-ca\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.855060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.849252 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-maas-api-tls\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.876066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.856839 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eecfa99-af26-4a92-9de5-dbf5afdb1609-openshift-service-ca\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.886079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.881733 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-maas-api-tls\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.887246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.887207 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmxv\" (UniqueName: \"kubernetes.io/projected/9eecfa99-af26-4a92-9de5-dbf5afdb1609-kube-api-access-xrmxv\") pod \"maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:01.992513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:01.992475 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:02.294900 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:02.294810 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth"] Nov 01 16:04:02.302363 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:04:02.302303 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eecfa99_af26_4a92_9de5_dbf5afdb1609.slice/crio-bf97eee6661d7e0c91b27c8d828b841627064da264d38341556049b877cb9b7f WatchSource:0}: Error finding container bf97eee6661d7e0c91b27c8d828b841627064da264d38341556049b877cb9b7f: Status 404 returned error can't find the container with id bf97eee6661d7e0c91b27c8d828b841627064da264d38341556049b877cb9b7f Nov 01 16:04:03.062768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:03.062716 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" event={"ID":"9eecfa99-af26-4a92-9de5-dbf5afdb1609","Type":"ContainerStarted","Data":"da9929f59de59cb8b1143fbe7a2ae59471276633dfc84b287b8f9d42251894ab"} Nov 01 16:04:03.063351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:03.062790 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" event={"ID":"9eecfa99-af26-4a92-9de5-dbf5afdb1609","Type":"ContainerStarted","Data":"bf97eee6661d7e0c91b27c8d828b841627064da264d38341556049b877cb9b7f"} Nov 01 16:04:03.063351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:03.063025 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:03.087350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:03.087285 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" podStartSLOduration=1.6744610199999999 podStartE2EDuration="2.087269474s" totalImagesPullingTime="412.808454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:03.083780172 +0000 UTC m=+983.228234813" watchObservedRunningTime="2026-11-01 16:04:03.087269474 +0000 UTC m=+983.231724134" Nov 01 16:04:09.097968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:09.097934 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:09.904065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:09.876445 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84"] Nov 01 16:04:10.115157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.115098 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.145272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.145240 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-47e1515a-serving-cert\"" Nov 01 16:04:10.227123 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.226756 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpbg\" (UniqueName: \"kubernetes.io/projected/38bad900-b43d-4369-a023-d3989c34b892-kube-api-access-ccpbg\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.227123 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.226888 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38bad900-b43d-4369-a023-d3989c34b892-openshift-service-ca\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.227123 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.227008 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-metrics-certs\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.227419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.227176 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.333845 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-metrics-certs\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.334086 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.334261 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpbg\" (UniqueName: \"kubernetes.io/projected/38bad900-b43d-4369-a023-d3989c34b892-kube-api-access-ccpbg\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.334382 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38bad900-b43d-4369-a023-d3989c34b892-openshift-service-ca\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.335620 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38bad900-b43d-4369-a023-d3989c34b892-openshift-service-ca\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:04:10.336192 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-serving-cert: secret "maas-api-e2e-mt-47e1515a-serving-cert" not found Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:04:10.336282 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls podName:38bad900-b43d-4369-a023-d3989c34b892 nodeName:}" failed. No retries permitted until 2026-11-01 16:04:10.83625822 +0000 UTC m=+990.980712851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls") pod "maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" (UID: "38bad900-b43d-4369-a023-d3989c34b892") : secret "maas-api-e2e-mt-47e1515a-serving-cert" not found Nov 01 16:04:10.368071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.361189 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpbg\" (UniqueName: \"kubernetes.io/projected/38bad900-b43d-4369-a023-d3989c34b892-kube-api-access-ccpbg\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.379735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.379697 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-metrics-certs\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.404192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.404110 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84"] Nov 01 16:04:10.881409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.881364 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:10.928667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:10.928622 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls\") pod \"maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:11.028061 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:04:11.023397 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:04:11.049274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.049237 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:11.248650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.248605 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.516129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.355431 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.710494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.621023 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.710494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.628080 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.710494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.628915 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:04:11.710494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.658619 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.710494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.674508 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:04:11.879832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.790218 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.879832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.815848 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:11.919562 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:11.893604 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.157253 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.157133 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.222066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.176574 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:04:12.367096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.273503 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.367096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.273848 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:04:12.367096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.301388 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:04:12.478611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.378169 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:04:12.532015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.530941 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.532015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.531853 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:04:12.570373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.570339 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84"] Nov 01 16:04:12.629124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.629082 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.659729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.630908 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:04:12.659729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.633097 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.659729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.659667 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:04:12.662671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.662634 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.741279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.678154 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.741279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.733237 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:04:12.741279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.736551 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:04:12.741279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.736613 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:04:12.824065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.733518 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 01 16:04:12.824065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.791318 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.824065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.791738 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:04:12.824065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.792501 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.824065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.794212 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:04:12.824065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.794294 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:04:12.899472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.843488 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.899472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:12.865408 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.171183 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.171712 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.235164 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.236357 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.264621 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.264688 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.280206 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.316121 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.348236 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.350059 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:04:13.373130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.350127 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:04:13.510607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.482704 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" event={"ID":"38bad900-b43d-4369-a023-d3989c34b892","Type":"ContainerStarted","Data":"31395196bd3a4e414d7c1969c3264da4d6113af4f95c23c979d1e6f929f8a54b"} Nov 01 16:04:13.590236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.556996 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.634315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.634201 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.636152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.636114 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.786281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.742695 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.786281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.748280 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:04:13.802992 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.802949 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:04:13.816512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.803292 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.816756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.806209 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:04:13.852002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:13.851954 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.296062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.291149 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.296062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.291530 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:04:14.296062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.291826 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:04:14.536073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.533379 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.594067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.562173 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:04:14.594067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.565530 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:04:14.594067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.568356 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.823065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.822092 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.838072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.828496 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.873243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.872073 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:04:14.894098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.886485 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:04:14.894098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.889683 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 01 16:04:14.919072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.916881 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.952522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.937263 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.007368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:14.988133 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.053310 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:04:15.028292 4911 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.054s" Nov 01 16:04:15.300165 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.297711 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.302162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.302108 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:04:15.314582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.314541 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:04:15.540204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.540154 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:04:15.580266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.580147 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:04:15.581700 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.581665 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:04:15.696748 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.696532 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:15.885161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.872218 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.885161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.784793 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.756255 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded" Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.697876 4911 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.900494 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.839254 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.788121 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.839165 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.839205 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:04:15.908724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.901712 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:04:15.911655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.911604 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:04:15.916066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.912173 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:04:15.959207 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:15.959144 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:04:16.024064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.000873 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:04:16.099028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.098974 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.361180 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.224229 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.451497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.451384 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:04:16.452859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.452823 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:04:16.561175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.544412 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.598878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.598841 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.643396 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.643349 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.693868 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.693828 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:04:16.933062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.933003 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.945065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.933674 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 01 16:04:16.948524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.948488 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:04:16.966514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.956127 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:04:16.969719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:16.966104 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.016213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.013199 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.367131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.366185 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.401066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.381683 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:04:17.454168 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.453147 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:04:17.554057 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.553992 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:04:17.580863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.578944 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:04:17.607020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.592204 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:04:17.830370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.830322 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:04:17.869550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:17.869503 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.009187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.009133 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.022984 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.022136 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.042063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.037276 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.077066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.044256 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.172408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.172298 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:04:18.224497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.202594 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.304248 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.287023 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.396473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.396425 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:04:18.794280 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.635273 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:04:18.798403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.664881 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.808067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.710510 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" podStartSLOduration=7.982619111 podStartE2EDuration="9.710481547s" totalImagesPullingTime="1.727862436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:18.66326635 +0000 UTC m=+998.807720992" watchObservedRunningTime="2026-11-01 16:04:18.710481547 +0000 UTC m=+998.854936187" Nov 01 16:04:18.817939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.710590 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.817939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.792553 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882340 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882392 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" event={"ID":"38bad900-b43d-4369-a023-d3989c34b892","Type":"ContainerStarted","Data":"15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8"} Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882467 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882495 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882559 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882607 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882655 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:04:18.885065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:18.882707 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:04:19.043689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.043641 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.428425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.424622 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.452217 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.452180 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:04:19.548067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.485448 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.742202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.676500 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:04:19.770092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.687023 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:04:19.770425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.770399 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:04:19.770678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.751685 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.814667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.814622 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.815868 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:19.815840 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:04:20.266105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:20.266074 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:04:20.266356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:20.266342 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:04:20.481441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:20.481391 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:04:24.738768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:24.738730 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:04:27.760423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.760384 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x"] Nov 01 16:04:27.765056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.765010 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.773512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.773477 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 16:04:27.773959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.773929 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 16:04:27.774778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.774754 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 16:04:27.775120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.774755 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k9xjt\"" Nov 01 16:04:27.806875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.806837 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x"] Nov 01 16:04:27.834897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.834860 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9g57\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-kube-api-access-v9g57\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.835162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.835083 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.835162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.835158 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.835306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.835194 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.835377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.835316 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.835377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.835364 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.835487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.835438 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.937318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.937280 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.937512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.937342 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.937512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.937424 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.937512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.937460 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9g57\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-kube-api-access-v9g57\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.937685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.937607 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.937685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.937678 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.937814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.937715 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.938140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.938111 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.939907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.939879 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.940146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.940121 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.940210 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.940188 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.940253 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.940211 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.947507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.947480 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:27.947819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:27.947787 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9g57\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-kube-api-access-v9g57\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2qs8x\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:28.096611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:28.096202 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:28.411367 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:28.411328 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x"] Nov 01 16:04:28.411498 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:04:28.411391 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61ca4b5_2b3b_4017_adfb_a6f70e1c4de5.slice/crio-1235db931af85cd067e8140d4f7124b4adb336219ee2220ea05ea2b1434cc4e1 WatchSource:0}: Error finding container 1235db931af85cd067e8140d4f7124b4adb336219ee2220ea05ea2b1434cc4e1: Status 404 returned error can't find the container with id 1235db931af85cd067e8140d4f7124b4adb336219ee2220ea05ea2b1434cc4e1 Nov 01 16:04:28.857237 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:28.854186 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" event={"ID":"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5","Type":"ContainerStarted","Data":"1235db931af85cd067e8140d4f7124b4adb336219ee2220ea05ea2b1434cc4e1"} Nov 01 16:04:32.360246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:32.360213 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:32.360553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:32.360281 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:32.944422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:32.944378 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" event={"ID":"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5","Type":"ContainerStarted","Data":"349ee28e395a595dd49418a4b0e0ac47ad424995f1700bd2b4f748d44660539e"} Nov 01 16:04:32.944756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:32.944739 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:32.972286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:32.972226 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" podStartSLOduration=2.026902349 podStartE2EDuration="5.972210034s" totalImagesPullingTime="3.945307685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:32.967998294 +0000 UTC m=+1013.112452936" watchObservedRunningTime="2026-11-01 16:04:32.972210034 +0000 UTC m=+1013.116664675" Nov 01 16:04:33.969965 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:33.969935 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:04:39.533756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:39.533719 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth"] Nov 01 16:04:39.534711 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:39.534655 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" podUID="9eecfa99-af26-4a92-9de5-dbf5afdb1609" containerName="maas-api" containerID="cri-o://da9929f59de59cb8b1143fbe7a2ae59471276633dfc84b287b8f9d42251894ab" gracePeriod=30 Nov 01 16:04:40.140201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.137581 4911 generic.go:356] "Generic (PLEG): container finished" podID="9eecfa99-af26-4a92-9de5-dbf5afdb1609" containerID="da9929f59de59cb8b1143fbe7a2ae59471276633dfc84b287b8f9d42251894ab" exitCode=0 Nov 01 16:04:40.463116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.461715 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp"] Nov 01 16:04:40.463116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.461795 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" event={"ID":"9eecfa99-af26-4a92-9de5-dbf5afdb1609","Type":"ContainerDied","Data":"da9929f59de59cb8b1143fbe7a2ae59471276633dfc84b287b8f9d42251894ab"} Nov 01 16:04:40.508127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.469560 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" podUID="30afe6eb-2bc1-42a8-b21a-ab067ac95831" containerName="istio-proxy" containerID="cri-o://748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69" gracePeriod=30 Nov 01 16:04:40.508127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.484246 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:40.525908 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.525869 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-metrics-certs\") pod \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " Nov 01 16:04:40.526791 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.526765 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eecfa99-af26-4a92-9de5-dbf5afdb1609-kube-api-access-xrmxv\" (UniqueName: \"kubernetes.io/projected/9eecfa99-af26-4a92-9de5-dbf5afdb1609-kube-api-access-xrmxv\") pod \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " Nov 01 16:04:40.527361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.527342 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-maas-api-tls\") pod \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " Nov 01 16:04:40.759859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.743363 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eecfa99-af26-4a92-9de5-dbf5afdb1609-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eecfa99-af26-4a92-9de5-dbf5afdb1609-openshift-service-ca\") pod \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\" (UID: \"9eecfa99-af26-4a92-9de5-dbf5afdb1609\") " Nov 01 16:04:40.759859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.736374 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eecfa99-af26-4a92-9de5-dbf5afdb1609-kube-api-access-xrmxv" pod "9eecfa99-af26-4a92-9de5-dbf5afdb1609" (UID: "9eecfa99-af26-4a92-9de5-dbf5afdb1609"). InnerVolumeSpecName "kube-api-access-xrmxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:40.759859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.736441 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-maas-api-tls" pod "9eecfa99-af26-4a92-9de5-dbf5afdb1609" (UID: "9eecfa99-af26-4a92-9de5-dbf5afdb1609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:40.759859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.746793 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eecfa99-af26-4a92-9de5-dbf5afdb1609-openshift-service-ca" pod "9eecfa99-af26-4a92-9de5-dbf5afdb1609" (UID: "9eecfa99-af26-4a92-9de5-dbf5afdb1609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:40.774721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.767112 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-metrics-certs" pod "9eecfa99-af26-4a92-9de5-dbf5afdb1609" (UID: "9eecfa99-af26-4a92-9de5-dbf5afdb1609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:40.846011 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.845892 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmxv\" (UniqueName: \"kubernetes.io/projected/9eecfa99-af26-4a92-9de5-dbf5afdb1609-kube-api-access-xrmxv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:40.846011 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.845935 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:40.846011 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.845953 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eecfa99-af26-4a92-9de5-dbf5afdb1609-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:40.846011 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:40.845967 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eecfa99-af26-4a92-9de5-dbf5afdb1609-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:40.893311 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:04:40.893229 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:04:41.183299 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:41.183199 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" event={"ID":"9eecfa99-af26-4a92-9de5-dbf5afdb1609","Type":"ContainerDied","Data":"bf97eee6661d7e0c91b27c8d828b841627064da264d38341556049b877cb9b7f"} Nov 01 16:04:41.183587 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:41.183566 4911 scope.go:122] "RemoveContainer" containerID="da9929f59de59cb8b1143fbe7a2ae59471276633dfc84b287b8f9d42251894ab" Nov 01 16:04:41.183871 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:41.183854 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth" Nov 01 16:04:41.261065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:41.257610 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth"] Nov 01 16:04:41.272027 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:41.271341 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3664624a-64f55f5b98-ztcth"] Nov 01 16:04:41.978197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:41.978161 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eecfa99-af26-4a92-9de5-dbf5afdb1609" path="/var/lib/kubelet/pods/9eecfa99-af26-4a92-9de5-dbf5afdb1609/volumes" Nov 01 16:04:45.935103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:45.935035 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:04:46.109551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.109507 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-credential-socket\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.109812 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.109790 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-socket\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.110244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.109910 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-credential-socket" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:46.110244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110004 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-certs\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.110244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110161 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-token\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-token\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.110244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110217 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-data\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.110502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110246 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-podinfo\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.110502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110309 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-kube-api-access-n9wlt\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-kube-api-access-n9wlt\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.110502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110446 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-envoy\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.110502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110475 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istiod-ca-cert\") pod \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\" (UID: \"30afe6eb-2bc1-42a8-b21a-ab067ac95831\") " Nov 01 16:04:46.111493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.111469 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.112290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.110238 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-socket" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:46.112417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.111776 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-certs" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:46.112522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.111872 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istiod-ca-cert" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:46.112607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.112264 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-data" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:46.126292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.126249 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-envoy" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:46.128489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.128442 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-podinfo" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:04:46.128489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.128479 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-token" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:46.130134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.130095 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-kube-api-access-n9wlt" pod "30afe6eb-2bc1-42a8-b21a-ab067ac95831" (UID: "30afe6eb-2bc1-42a8-b21a-ab067ac95831"). InnerVolumeSpecName "kube-api-access-n9wlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:46.212898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.212864 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.213115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.213101 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.213196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.213186 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.213265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.213257 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.213331 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.213319 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.213383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.213374 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9wlt\" (UniqueName: \"kubernetes.io/projected/30afe6eb-2bc1-42a8-b21a-ab067ac95831-kube-api-access-n9wlt\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.213441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.213434 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.213503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.213495 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30afe6eb-2bc1-42a8-b21a-ab067ac95831-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:46.545060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.544061 4911 generic.go:356] "Generic (PLEG): container finished" podID="30afe6eb-2bc1-42a8-b21a-ab067ac95831" containerID="748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69" exitCode=0 Nov 01 16:04:46.545060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.544390 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" event={"ID":"30afe6eb-2bc1-42a8-b21a-ab067ac95831","Type":"ContainerDied","Data":"748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69"} Nov 01 16:04:46.545060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.544465 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" event={"ID":"30afe6eb-2bc1-42a8-b21a-ab067ac95831","Type":"ContainerDied","Data":"25471b8e8e143f17da6b0225f4b9793c26c0bf652a2021d22c85141035bd38cb"} Nov 01 16:04:46.545060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.544499 4911 scope.go:122] "RemoveContainer" containerID="748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69" Nov 01 16:04:46.545060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.544700 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp" Nov 01 16:04:46.639130 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.638093 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp"] Nov 01 16:04:46.794521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.771241 4911 scope.go:122] "RemoveContainer" containerID="748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69" Nov 01 16:04:46.794521 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:04:46.771718 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69\": container with ID starting with 748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69 not found: ID does not exist" containerID="748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69" Nov 01 16:04:46.794521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.771757 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69"} err="failed to get container status \"748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69\": rpc error: code = NotFound desc = could not find container \"748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69\": container with ID starting with 748dc42daa5963175c91105e4823e328cdc48cf19d120647a4158696ba09fe69 not found: ID does not exist" Nov 01 16:04:46.808823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:46.801487 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3664624a-gw-openshift-default-7ff9b75cb5-dgsdp"] Nov 01 16:04:48.001540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:48.001505 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30afe6eb-2bc1-42a8-b21a-ab067ac95831" path="/var/lib/kubelet/pods/30afe6eb-2bc1-42a8-b21a-ab067ac95831/volumes" Nov 01 16:04:54.746651 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:54.746607 4911 generic.go:356] "Generic (PLEG): container finished" podID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerID="8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09" exitCode=0 Nov 01 16:04:54.747161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:54.746675 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" event={"ID":"e92ffd17-2932-46d4-96fa-8b083ba5ebdc","Type":"ContainerDied","Data":"8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09"} Nov 01 16:04:54.747325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:54.747302 4911 scope.go:122] "RemoveContainer" containerID="8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09" Nov 01 16:04:54.749287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:54.749253 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:54.749400 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:54.749345 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:54.749400 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:54.749393 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:55.771002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:55.770958 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" event={"ID":"e92ffd17-2932-46d4-96fa-8b083ba5ebdc","Type":"ContainerStarted","Data":"3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d"} Nov 01 16:04:55.927432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:55.927395 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:04:55.927833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:55.927733 4911 patch_prober.go:29] interesting pod/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 16:04:55.928066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:55.928013 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 16:04:56.925992 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:56.925956 4911 patch_prober.go:29] interesting pod/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 16:04:56.926486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:56.926023 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 16:04:57.926245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:57.926207 4911 patch_prober.go:29] interesting pod/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 16:04:57.926648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:57.926275 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 16:04:58.925925 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:58.925886 4911 patch_prober.go:29] interesting pod/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 16:04:58.926140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:58.925944 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 16:04:59.925855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:59.925807 4911 patch_prober.go:29] interesting pod/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 16:04:59.926388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:04:59.925881 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 16:05:00.925585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:00.925537 4911 patch_prober.go:29] interesting pod/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 16:05:00.925800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:00.925618 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 16:05:01.041235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:01.041194 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:05:01.930473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:01.930439 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:05:01.931706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:01.931680 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:05:10.048932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:10.048867 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb"] Nov 01 16:05:10.061671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:10.061624 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218dc07-jfjnb"] Nov 01 16:05:10.874361 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:05:10.874306 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:05:11.967387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:11.967350 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e31a76-a70f-4d9c-825a-9e141962cbca" path="/var/lib/kubelet/pods/84e31a76-a70f-4d9c-825a-9e141962cbca/volumes" Nov 01 16:05:16.025310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:16.025269 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5"] Nov 01 16:05:16.028406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:16.028367 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d29cbb63-p4wz5"] Nov 01 16:05:18.002916 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:18.002874 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fe9127-45e5-4751-b211-65f91a376efe" path="/var/lib/kubelet/pods/48fe9127-45e5-4751-b211-65f91a376efe/volumes" Nov 01 16:05:21.073946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:21.073909 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq"] Nov 01 16:05:21.080652 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:21.080625 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8f35e65-b4bcq"] Nov 01 16:05:21.986451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:21.986410 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792d8743-937e-4de1-b29e-48b04c55e69c" path="/var/lib/kubelet/pods/792d8743-937e-4de1-b29e-48b04c55e69c/volumes" Nov 01 16:05:26.034607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:26.034564 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk"] Nov 01 16:05:26.041187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:26.041152 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeac48cb-bnwkk"] Nov 01 16:05:27.969308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:27.969267 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1e9534-870d-431f-b22d-935f969c3e45" path="/var/lib/kubelet/pods/ac1e9534-870d-431f-b22d-935f969c3e45/volumes" Nov 01 16:05:40.872267 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:05:40.872189 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:05:50.840591 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.840555 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv"] Nov 01 16:05:50.841236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.841220 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9eecfa99-af26-4a92-9de5-dbf5afdb1609" containerName="maas-api" Nov 01 16:05:50.841236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.841237 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eecfa99-af26-4a92-9de5-dbf5afdb1609" containerName="maas-api" Nov 01 16:05:50.841326 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.841286 4911 cpu_manager.go:414] "containerMap: removing container" podUID="30afe6eb-2bc1-42a8-b21a-ab067ac95831" containerName="istio-proxy" Nov 01 16:05:50.841326 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.841291 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30afe6eb-2bc1-42a8-b21a-ab067ac95831" containerName="istio-proxy" Nov 01 16:05:50.841413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.841403 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="30afe6eb-2bc1-42a8-b21a-ab067ac95831" containerName="istio-proxy" Nov 01 16:05:50.841451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.841418 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eecfa99-af26-4a92-9de5-dbf5afdb1609" containerName="maas-api" Nov 01 16:05:50.846468 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.846393 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:50.853277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.853250 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv"] Nov 01 16:05:50.874712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:50.874676 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c8d121-openshift-default-dockercfg-rn95k\"" Nov 01 16:05:51.059065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.042904 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhqzc\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-kube-api-access-zhqzc\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.073065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.060372 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-data\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.073511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.073479 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-certs\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.073734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.073702 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-credential-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.073962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.073946 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-podinfo\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.074166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.074150 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.074367 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.074350 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-envoy\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.074512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.074496 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.074716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.074697 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-token\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.185866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.185765 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-data\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.186672 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.186646 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-certs\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.187469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.187191 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-credential-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.193501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.187108 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-certs\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.193501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.186489 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-data\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.193501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.187442 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-credential-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.194332 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.194307 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-podinfo\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.194642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.194616 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.203674 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.195034 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-socket\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.206329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.206303 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-envoy\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.214059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.206931 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.215532 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.215502 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-token\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.230495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.215032 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.230495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.224555 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-podinfo\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.230495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.227489 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-envoy\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.230495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.229749 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-token\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.230495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.230272 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhqzc\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-kube-api-access-zhqzc\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.244936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.244901 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhqzc\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-kube-api-access-zhqzc\") pod \"e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.477988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.477892 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:51.782457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.782430 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv"] Nov 01 16:05:51.782875 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:05:51.782847 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74269e5_3eee_4c31_acf1_a610e61b8ed3.slice/crio-33ce68b5b34b0d7b1f2cd216c5027540152fbc73ab3c7058009b442b59618f9e WatchSource:0}: Error finding container 33ce68b5b34b0d7b1f2cd216c5027540152fbc73ab3c7058009b442b59618f9e: Status 404 returned error can't find the container with id 33ce68b5b34b0d7b1f2cd216c5027540152fbc73ab3c7058009b442b59618f9e Nov 01 16:05:51.785614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.785580 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:51.785687 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.785654 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:51.785731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:51.785685 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:52.619272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:52.619229 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" event={"ID":"a74269e5-3eee-4c31-acf1-a610e61b8ed3","Type":"ContainerStarted","Data":"ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45"} Nov 01 16:05:52.619866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:52.619322 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" event={"ID":"a74269e5-3eee-4c31-acf1-a610e61b8ed3","Type":"ContainerStarted","Data":"33ce68b5b34b0d7b1f2cd216c5027540152fbc73ab3c7058009b442b59618f9e"} Nov 01 16:05:52.642872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:52.642814 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podStartSLOduration=2.6427984970000002 podStartE2EDuration="2.642798497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:52.638943009 +0000 UTC m=+1092.783397651" watchObservedRunningTime="2026-11-01 16:05:52.642798497 +0000 UTC m=+1092.787253137" Nov 01 16:05:53.478083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:53.478035 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:53.484365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:53.484333 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:53.626956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:53.626923 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:53.627999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:53.627979 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:05:56.147770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.147730 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn"] Nov 01 16:05:56.152246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.152221 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:56.161689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.161653 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c8d121\"" Nov 01 16:05:56.165502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.165478 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn"] Nov 01 16:05:56.345855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.345823 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbz4l\" (UniqueName: \"kubernetes.io/projected/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-kube-api-access-qbz4l\") pod \"payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:56.346166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.346131 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:56.464772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.460992 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:56.464772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.461250 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbz4l\" (UniqueName: \"kubernetes.io/projected/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-kube-api-access-qbz4l\") pod \"payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:56.464772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.463940 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:56.487083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.485928 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbz4l\" (UniqueName: \"kubernetes.io/projected/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-kube-api-access-qbz4l\") pod \"payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:56.771631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:56.771599 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:57.062238 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:05:57.062197 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0631e6_9b1d_4b0d_b99c_96036c6e0f0e.slice/crio-c84041df5992f585c21fe11c82783703139a89c6ba4e17774f4edad33b74cf17 WatchSource:0}: Error finding container c84041df5992f585c21fe11c82783703139a89c6ba4e17774f4edad33b74cf17: Status 404 returned error can't find the container with id c84041df5992f585c21fe11c82783703139a89c6ba4e17774f4edad33b74cf17 Nov 01 16:05:57.062709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:57.062686 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn"] Nov 01 16:05:57.668349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:57.668312 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" event={"ID":"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e","Type":"ContainerStarted","Data":"8f2492310eb2284af0f0727f6da76104c37ce9aca975a7396427ef0628a505c4"} Nov 01 16:05:57.668742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:57.668382 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:05:57.668742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:57.668410 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" event={"ID":"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e","Type":"ContainerStarted","Data":"c84041df5992f585c21fe11c82783703139a89c6ba4e17774f4edad33b74cf17"} Nov 01 16:05:57.686701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:57.686653 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" podStartSLOduration=1.68664155 podStartE2EDuration="1.68664155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:57.684198403 +0000 UTC m=+1097.828653045" watchObservedRunningTime="2026-11-01 16:05:57.68664155 +0000 UTC m=+1097.831096191" Nov 01 16:05:59.050584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.050547 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq"] Nov 01 16:05:59.055204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.055174 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.068740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.068711 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-a9a967-serving-cert\"" Nov 01 16:05:59.071120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.071073 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq"] Nov 01 16:05:59.104768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.104724 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxml4\" (UniqueName: \"kubernetes.io/projected/498dc930-9369-454f-83bd-b61c60db83ad-kube-api-access-hxml4\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.104930 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.104877 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.105101 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.105082 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.105248 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.105230 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/498dc930-9369-454f-83bd-b61c60db83ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.206399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.206365 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.206588 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.206491 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/498dc930-9369-454f-83bd-b61c60db83ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.206588 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.206550 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxml4\" (UniqueName: \"kubernetes.io/projected/498dc930-9369-454f-83bd-b61c60db83ad-kube-api-access-hxml4\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.206696 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.206637 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.207217 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.207189 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/498dc930-9369-454f-83bd-b61c60db83ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.208937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.208913 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.209081 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.209026 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.214736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.214711 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxml4\" (UniqueName: \"kubernetes.io/projected/498dc930-9369-454f-83bd-b61c60db83ad-kube-api-access-hxml4\") pod \"maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.379853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.379590 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:05:59.675389 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:05:59.674157 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498dc930_9369_454f_83bd_b61c60db83ad.slice/crio-556239102a99a89b10de002a1d7386f2a6c6d72521ed960e6b14ed37fc0eb7fe WatchSource:0}: Error finding container 556239102a99a89b10de002a1d7386f2a6c6d72521ed960e6b14ed37fc0eb7fe: Status 404 returned error can't find the container with id 556239102a99a89b10de002a1d7386f2a6c6d72521ed960e6b14ed37fc0eb7fe Nov 01 16:05:59.686423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.686399 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq"] Nov 01 16:05:59.692614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:05:59.692584 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" event={"ID":"498dc930-9369-454f-83bd-b61c60db83ad","Type":"ContainerStarted","Data":"556239102a99a89b10de002a1d7386f2a6c6d72521ed960e6b14ed37fc0eb7fe"} Nov 01 16:06:00.702830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:00.702786 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" event={"ID":"498dc930-9369-454f-83bd-b61c60db83ad","Type":"ContainerStarted","Data":"274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef"} Nov 01 16:06:00.703364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:00.702898 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:06:00.725154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:00.725097 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" podStartSLOduration=1.292337254 podStartE2EDuration="1.725080726s" totalImagesPullingTime="432.743472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:00.720762566 +0000 UTC m=+1100.865217208" watchObservedRunningTime="2026-11-01 16:06:00.725080726 +0000 UTC m=+1100.869535368" Nov 01 16:06:03.513087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.513010 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb"] Nov 01 16:06:03.520535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.518175 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.527252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.527220 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb"] Nov 01 16:06:03.529174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.529122 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c8d121-serving-cert\"" Nov 01 16:06:03.604979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.604937 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.605192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.605167 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9fz\" (UniqueName: \"kubernetes.io/projected/c2464d8c-3a41-4040-9087-159740306188-kube-api-access-jj9fz\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.605311 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.605289 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.605484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.605456 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2464d8c-3a41-4040-9087-159740306188-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.706884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.706842 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.707067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.706967 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9fz\" (UniqueName: \"kubernetes.io/projected/c2464d8c-3a41-4040-9087-159740306188-kube-api-access-jj9fz\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.707067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.707026 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.707186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.707105 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2464d8c-3a41-4040-9087-159740306188-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.707788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.707767 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2464d8c-3a41-4040-9087-159740306188-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.709539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.709516 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.709695 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.709670 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.715779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.715753 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9fz\" (UniqueName: \"kubernetes.io/projected/c2464d8c-3a41-4040-9087-159740306188-kube-api-access-jj9fz\") pod \"maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:03.857580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:03.857474 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:04.298410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:04.298376 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb"] Nov 01 16:06:04.748002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:04.747957 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" event={"ID":"c2464d8c-3a41-4040-9087-159740306188","Type":"ContainerStarted","Data":"8878811743e966d3072f653a23b0a4ecc6e33b41006e08d63428c959225239f0"} Nov 01 16:06:05.758711 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:05.758667 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" event={"ID":"c2464d8c-3a41-4040-9087-159740306188","Type":"ContainerStarted","Data":"f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3"} Nov 01 16:06:05.759128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:05.758757 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:05.779127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:05.779077 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" podStartSLOduration=2.321661936 podStartE2EDuration="2.779065477s" totalImagesPullingTime="457.403541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:05.775328128 +0000 UTC m=+1105.919782769" watchObservedRunningTime="2026-11-01 16:06:05.779065477 +0000 UTC m=+1105.923520117" Nov 01 16:06:06.717205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:06.717170 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:06:08.681815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:08.681784 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:06:10.870097 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:06:10.870020 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:06:11.773836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:11.773804 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:06:21.581865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.581666 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84"] Nov 01 16:06:21.582656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.582594 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" podUID="38bad900-b43d-4369-a023-d3989c34b892" containerName="maas-api" containerID="cri-o://15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8" gracePeriod=30 Nov 01 16:06:21.961116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.961017 4911 generic.go:356] "Generic (PLEG): container finished" podID="38bad900-b43d-4369-a023-d3989c34b892" containerID="15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8" exitCode=0 Nov 01 16:06:21.962727 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.962423 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:06:21.968788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.968502 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" event={"ID":"38bad900-b43d-4369-a023-d3989c34b892","Type":"ContainerDied","Data":"15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8"} Nov 01 16:06:21.968788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.968558 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" event={"ID":"38bad900-b43d-4369-a023-d3989c34b892","Type":"ContainerDied","Data":"31395196bd3a4e414d7c1969c3264da4d6113af4f95c23c979d1e6f929f8a54b"} Nov 01 16:06:21.968788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.968594 4911 scope.go:122] "RemoveContainer" containerID="15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8" Nov 01 16:06:21.987266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.985963 4911 scope.go:122] "RemoveContainer" containerID="15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8" Nov 01 16:06:21.987266 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:06:21.986437 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8\": container with ID starting with 15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8 not found: ID does not exist" containerID="15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8" Nov 01 16:06:21.987266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:21.986473 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8"} err="failed to get container status \"15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8\": rpc error: code = NotFound desc = could not find container \"15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8\": container with ID starting with 15cb591b3bb38b63714b53e94b028999f0d70157d9894762d9b0499b39edb5c8 not found: ID does not exist" Nov 01 16:06:22.097060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.092534 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bad900-b43d-4369-a023-d3989c34b892-kube-api-access-ccpbg\" (UniqueName: \"kubernetes.io/projected/38bad900-b43d-4369-a023-d3989c34b892-kube-api-access-ccpbg\") pod \"38bad900-b43d-4369-a023-d3989c34b892\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " Nov 01 16:06:22.097060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.092657 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-metrics-certs\") pod \"38bad900-b43d-4369-a023-d3989c34b892\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " Nov 01 16:06:22.097060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.092893 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bad900-b43d-4369-a023-d3989c34b892-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38bad900-b43d-4369-a023-d3989c34b892-openshift-service-ca\") pod \"38bad900-b43d-4369-a023-d3989c34b892\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " Nov 01 16:06:22.097060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.092947 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls\") pod \"38bad900-b43d-4369-a023-d3989c34b892\" (UID: \"38bad900-b43d-4369-a023-d3989c34b892\") " Nov 01 16:06:22.105806 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.099239 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bad900-b43d-4369-a023-d3989c34b892-openshift-service-ca" pod "38bad900-b43d-4369-a023-d3989c34b892" (UID: "38bad900-b43d-4369-a023-d3989c34b892"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:06:22.177960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.168721 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bad900-b43d-4369-a023-d3989c34b892-kube-api-access-ccpbg" pod "38bad900-b43d-4369-a023-d3989c34b892" (UID: "38bad900-b43d-4369-a023-d3989c34b892"). InnerVolumeSpecName "kube-api-access-ccpbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:22.177960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.171034 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-metrics-certs" pod "38bad900-b43d-4369-a023-d3989c34b892" (UID: "38bad900-b43d-4369-a023-d3989c34b892"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:06:22.177960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.177390 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls" pod "38bad900-b43d-4369-a023-d3989c34b892" (UID: "38bad900-b43d-4369-a023-d3989c34b892"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:06:22.195448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.195417 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpbg\" (UniqueName: \"kubernetes.io/projected/38bad900-b43d-4369-a023-d3989c34b892-kube-api-access-ccpbg\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:22.195448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.195449 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:22.195640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.195463 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38bad900-b43d-4369-a023-d3989c34b892-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:22.195640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.195472 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38bad900-b43d-4369-a023-d3989c34b892-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:22.970345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.970314 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84" Nov 01 16:06:22.998534 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:22.998502 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84"] Nov 01 16:06:23.004343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:23.004315 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47e1515a-55cdcf8557-fkv84"] Nov 01 16:06:23.973601 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:23.970959 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bad900-b43d-4369-a023-d3989c34b892" path="/var/lib/kubelet/pods/38bad900-b43d-4369-a023-d3989c34b892/volumes" Nov 01 16:06:26.515156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.515118 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4"] Nov 01 16:06:26.515870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.515850 4911 cpu_manager.go:414] "containerMap: removing container" podUID="38bad900-b43d-4369-a023-d3989c34b892" containerName="maas-api" Nov 01 16:06:26.515870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.515869 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bad900-b43d-4369-a023-d3989c34b892" containerName="maas-api" Nov 01 16:06:26.516087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.516068 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bad900-b43d-4369-a023-d3989c34b892" containerName="maas-api" Nov 01 16:06:26.522094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.522029 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.528414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.528390 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-76063d10-serving-cert\"" Nov 01 16:06:26.536518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.536489 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4"] Nov 01 16:06:26.668553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.668522 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842j6\" (UniqueName: \"kubernetes.io/projected/0efa9f9d-1f9f-4612-877e-cf1819dab209-kube-api-access-842j6\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.668735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.668564 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-metrics-certs\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.668778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.668741 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efa9f9d-1f9f-4612-877e-cf1819dab209-openshift-service-ca\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.668778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.668772 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-maas-api-tls\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.770577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.770477 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efa9f9d-1f9f-4612-877e-cf1819dab209-openshift-service-ca\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.770577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.770553 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-maas-api-tls\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.770843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.770777 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842j6\" (UniqueName: \"kubernetes.io/projected/0efa9f9d-1f9f-4612-877e-cf1819dab209-kube-api-access-842j6\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.770843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.770813 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-metrics-certs\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.771369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.771331 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efa9f9d-1f9f-4612-877e-cf1819dab209-openshift-service-ca\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.773606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.773582 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-metrics-certs\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.773781 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.773759 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-maas-api-tls\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.779379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.779354 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-842j6\" (UniqueName: \"kubernetes.io/projected/0efa9f9d-1f9f-4612-877e-cf1819dab209-kube-api-access-842j6\") pod \"maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:26.848356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:26.848313 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:27.244124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:27.237054 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4"] Nov 01 16:06:28.031440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:28.031405 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" event={"ID":"0efa9f9d-1f9f-4612-877e-cf1819dab209","Type":"ContainerStarted","Data":"5d7ee57b412404553f3b8826ff12907dacf1243493b62299b9ab135ebc9f14a2"} Nov 01 16:06:28.031855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:28.031460 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" event={"ID":"0efa9f9d-1f9f-4612-877e-cf1819dab209","Type":"ContainerStarted","Data":"ad1152ae3f83dea78463853f993b3a01ca28bc9ef6131c71e0d95d3df0bfc2a2"} Nov 01 16:06:28.031855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:28.031489 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:28.055308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:28.055242 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" podStartSLOduration=1.625024885 podStartE2EDuration="2.05522296s" totalImagesPullingTime="430.198075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:28.052499071 +0000 UTC m=+1128.196953713" watchObservedRunningTime="2026-11-01 16:06:28.05522296 +0000 UTC m=+1128.199677605" Nov 01 16:06:29.081564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:29.081528 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4"] Nov 01 16:06:29.082250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:29.081792 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" containerID="cri-o://3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d" gracePeriod=30 Nov 01 16:06:31.300814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.300776 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv"] Nov 01 16:06:31.306055 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.306020 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.320235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.317334 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a9a967-openshift-default-dockercfg-8bwnt\"" Nov 01 16:06:31.327263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.327237 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv"] Nov 01 16:06:31.329352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.329320 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.329531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.329376 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.329531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.329408 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q9v8\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-kube-api-access-5q9v8\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.330595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.330561 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.330714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.330669 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.330847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.330831 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.330943 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.330924 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.331021 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.330960 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.331021 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.330991 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.432630 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432596 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.432809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432692 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.432809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432722 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.432809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432750 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432840 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432874 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432900 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q9v8\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-kube-api-access-5q9v8\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.432994 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.433069 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.433093 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.433168 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.433210 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.433591 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.433744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.433694 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.435352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.435325 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.435740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.435717 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.441783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.441751 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q9v8\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-kube-api-access-5q9v8\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.441907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.441886 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.640470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.640373 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:31.933805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.933711 4911 patch_prober.go:29] interesting pod/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:06:31.933805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.933776 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:06:31.977362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.977329 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv"] Nov 01 16:06:31.979405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.979362 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:31.979496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.979446 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:31.979496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:31.979489 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.067797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:32.067762 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" event={"ID":"5e6e888f-bcad-46f7-8c32-04b78cfedd6b","Type":"ContainerStarted","Data":"ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22"} Nov 01 16:06:32.067953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:32.067824 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" event={"ID":"5e6e888f-bcad-46f7-8c32-04b78cfedd6b","Type":"ContainerStarted","Data":"4ad8d7926e536b11c4f49e0241262f4676fa78924fc48eefae3d2be1ddc34f30"} Nov 01 16:06:32.094746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:32.094679 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" podStartSLOduration=1.094659997 podStartE2EDuration="1.094659997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:32.088388239 +0000 UTC m=+1132.232842885" watchObservedRunningTime="2026-11-01 16:06:32.094659997 +0000 UTC m=+1132.239114639" Nov 01 16:06:32.640475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:32.640437 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:33.642966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:33.642932 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 16:06:33.643477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:33.642986 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 16:06:34.045753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.045724 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:06:34.432434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.432407 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:06:34.513238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513198 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-podinfo\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513307 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-envoy\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513343 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-credential-socket\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513401 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-data\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513449 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-certs\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513622 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-socket\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513669 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-token\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-token\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513728 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istiod-ca-cert\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513773 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-kube-api-access-6j6lf\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-kube-api-access-6j6lf\") pod \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\" (UID: \"e92ffd17-2932-46d4-96fa-8b083ba5ebdc\") " Nov 01 16:06:34.513945 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513907 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-socket" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:06:34.514012 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.513992 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-certs" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:06:34.514091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514068 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-data" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:06:34.514300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514274 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istiod-ca-cert" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:06:34.514415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514394 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-credential-socket" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:06:34.514716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514687 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.514819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514718 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.514819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514733 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.514819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514745 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.514819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.514760 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.516432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.516405 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-envoy" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:06:34.516545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.516504 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-podinfo" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:06:34.516717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.516694 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-kube-api-access-6j6lf" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "kube-api-access-6j6lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:34.516784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.516764 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-token" pod "e92ffd17-2932-46d4-96fa-8b083ba5ebdc" (UID: "e92ffd17-2932-46d4-96fa-8b083ba5ebdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:34.615879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.615847 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.615879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.615878 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j6lf\" (UniqueName: \"kubernetes.io/projected/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-kube-api-access-6j6lf\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.615879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.615889 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.616156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.615898 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92ffd17-2932-46d4-96fa-8b083ba5ebdc-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:34.640991 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.640954 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 16:06:34.641178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:34.641026 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 16:06:35.117025 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.116990 4911 generic.go:356] "Generic (PLEG): container finished" podID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerID="3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d" exitCode=0 Nov 01 16:06:35.117025 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.117049 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" Nov 01 16:06:35.117542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.117069 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" event={"ID":"e92ffd17-2932-46d4-96fa-8b083ba5ebdc","Type":"ContainerDied","Data":"3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d"} Nov 01 16:06:35.117542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.117128 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4" event={"ID":"e92ffd17-2932-46d4-96fa-8b083ba5ebdc","Type":"ContainerDied","Data":"8b298c7170b96640ac65b4774bc12b9c8aa8066cd3d06d6711525275fa5ee90f"} Nov 01 16:06:35.117542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.117150 4911 scope.go:122] "RemoveContainer" containerID="3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d" Nov 01 16:06:35.135328 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.135308 4911 scope.go:122] "RemoveContainer" containerID="8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09" Nov 01 16:06:35.147259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.147217 4911 scope.go:122] "RemoveContainer" containerID="3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d" Nov 01 16:06:35.185212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.185169 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4"] Nov 01 16:06:35.188348 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:06:35.188312 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d\": container with ID starting with 3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d not found: ID does not exist" containerID="3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d" Nov 01 16:06:35.188522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.188376 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d"} err="failed to get container status \"3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d\": rpc error: code = NotFound desc = could not find container \"3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d\": container with ID starting with 3c2d6ce579aa136bd859d1d0858e55b10cec8eb2364b77afe5d7a8b1db54ac3d not found: ID does not exist" Nov 01 16:06:35.188522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.188406 4911 scope.go:122] "RemoveContainer" containerID="8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09" Nov 01 16:06:35.188935 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:06:35.188904 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09\": container with ID starting with 8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09 not found: ID does not exist" containerID="8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09" Nov 01 16:06:35.189224 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.189172 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09"} err="failed to get container status \"8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09\": rpc error: code = NotFound desc = could not find container \"8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09\": container with ID starting with 8c79b62b793d836f7428a96a97b8e2d844cf8dff9bf6d5b67f891eee109c8c09 not found: ID does not exist" Nov 01 16:06:35.189865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.189845 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-47e1515a-openshift-default-69dc94c849-7vsn4"] Nov 01 16:06:35.641096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.641025 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 16:06:35.641279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:35.641131 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 16:06:36.071076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:36.065234 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" path="/var/lib/kubelet/pods/e92ffd17-2932-46d4-96fa-8b083ba5ebdc/volumes" Nov 01 16:06:36.641277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:36.641241 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 16:06:36.641744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:36.641313 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 16:06:37.641825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:37.641370 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 01 16:06:37.641825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:37.641465 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 01 16:06:38.749347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:38.749310 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:38.750406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:38.750379 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:38.751287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:38.751266 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:06:40.448733 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.448699 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926"] Nov 01 16:06:40.449466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.449450 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" Nov 01 16:06:40.449530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.449467 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" Nov 01 16:06:40.449530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.449479 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" Nov 01 16:06:40.449530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.449484 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" Nov 01 16:06:40.449620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.449593 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" Nov 01 16:06:40.449620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.449601 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92ffd17-2932-46d4-96fa-8b083ba5ebdc" containerName="istio-proxy" Nov 01 16:06:40.454561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.454539 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.465718 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.465691 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f47890-dockercfg-9k9tc\"" Nov 01 16:06:40.471449 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.471423 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926"] Nov 01 16:06:40.530487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.530450 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f47890\"" Nov 01 16:06:40.618595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.618550 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e88ab70-e83d-4abd-95a9-96308c0382d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.653462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.653429 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdfdv\" (UniqueName: \"kubernetes.io/projected/3e88ab70-e83d-4abd-95a9-96308c0382d1-kube-api-access-rdfdv\") pod \"payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.755559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.755510 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e88ab70-e83d-4abd-95a9-96308c0382d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.755798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.755667 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdfdv\" (UniqueName: \"kubernetes.io/projected/3e88ab70-e83d-4abd-95a9-96308c0382d1-kube-api-access-rdfdv\") pod \"payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.756395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.756370 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e88ab70-e83d-4abd-95a9-96308c0382d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.767504 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.767475 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdfdv\" (UniqueName: \"kubernetes.io/projected/3e88ab70-e83d-4abd-95a9-96308c0382d1-kube-api-access-rdfdv\") pod \"payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.771206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.771187 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:40.879765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.879727 4911 scope.go:122] "RemoveContainer" containerID="f8ff99335288cf94265f01d5641c1e13d911fc531e19e8cfdef4a474b106edf4" Nov 01 16:06:40.887478 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:06:40.886275 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:06:40.915898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.912611 4911 scope.go:122] "RemoveContainer" containerID="61976262b7dbac595ad5ee342c45431b9403c92b6cd1ebc2a83f69b79957d6c6" Nov 01 16:06:40.944872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.944830 4911 scope.go:122] "RemoveContainer" containerID="4964358a36fcc9bdb3aeb37b3a87446e8472b17f0791cb93656e72553ead2d18" Nov 01 16:06:40.970309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:40.969567 4911 scope.go:122] "RemoveContainer" containerID="98f2007a8a2773dc05044e83fdbdcd4ec1bd217b2db7c7bb7966985bf958409a" Nov 01 16:06:41.105895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:41.105866 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926"] Nov 01 16:06:41.107351 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:06:41.107302 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e88ab70_e83d_4abd_95a9_96308c0382d1.slice/crio-bbbd124951879ca07765d998215efb34f0fae607241573239f693fc2a8d6e9f6 WatchSource:0}: Error finding container bbbd124951879ca07765d998215efb34f0fae607241573239f693fc2a8d6e9f6: Status 404 returned error can't find the container with id bbbd124951879ca07765d998215efb34f0fae607241573239f693fc2a8d6e9f6 Nov 01 16:06:41.222967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:41.222932 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" event={"ID":"3e88ab70-e83d-4abd-95a9-96308c0382d1","Type":"ContainerStarted","Data":"bbbd124951879ca07765d998215efb34f0fae607241573239f693fc2a8d6e9f6"} Nov 01 16:06:42.235638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:42.235597 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" event={"ID":"3e88ab70-e83d-4abd-95a9-96308c0382d1","Type":"ContainerStarted","Data":"d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51"} Nov 01 16:06:42.236124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:42.235855 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:42.253644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:42.253573 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" podStartSLOduration=2.253554175 podStartE2EDuration="2.253554175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:42.251308565 +0000 UTC m=+1142.395763288" watchObservedRunningTime="2026-11-01 16:06:42.253554175 +0000 UTC m=+1142.398008816" Nov 01 16:06:49.536437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.536394 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk"] Nov 01 16:06:49.541210 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.541180 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.545291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.545196 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f47890-serving-cert\"" Nov 01 16:06:49.555916 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.555887 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk"] Nov 01 16:06:49.625604 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.625569 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.625604 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.625603 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvq6\" (UniqueName: \"kubernetes.io/projected/663b1737-410d-414f-8e86-c50c45268313-kube-api-access-kpvq6\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.625818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.625766 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b1737-410d-414f-8e86-c50c45268313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.626090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.626067 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.729597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.728747 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b1737-410d-414f-8e86-c50c45268313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.729597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.729109 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.729597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.729179 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.729597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.729210 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvq6\" (UniqueName: \"kubernetes.io/projected/663b1737-410d-414f-8e86-c50c45268313-kube-api-access-kpvq6\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.731230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.730828 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b1737-410d-414f-8e86-c50c45268313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.739728 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.739700 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.740125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.740095 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvq6\" (UniqueName: \"kubernetes.io/projected/663b1737-410d-414f-8e86-c50c45268313-kube-api-access-kpvq6\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.742382 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.742344 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:49.874153 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:49.873928 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:50.251951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:50.246390 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk"] Nov 01 16:06:50.324374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:50.324337 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" event={"ID":"663b1737-410d-414f-8e86-c50c45268313","Type":"ContainerStarted","Data":"642fbe0d3ff9152bdfbb60987b3f3fc534c2142345288f08c1834e5e399221dc"} Nov 01 16:06:51.344084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:51.342608 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" event={"ID":"663b1737-410d-414f-8e86-c50c45268313","Type":"ContainerStarted","Data":"6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da"} Nov 01 16:06:51.344084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:51.342688 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:06:51.364434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:51.364382 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" podStartSLOduration=1.8180611290000002 podStartE2EDuration="2.364361709s" totalImagesPullingTime="546.30058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:51.363980906 +0000 UTC m=+1151.508435547" watchObservedRunningTime="2026-11-01 16:06:51.364361709 +0000 UTC m=+1151.508816354" Nov 01 16:06:53.249839 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:53.249807 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:06:57.357881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:06:57.357850 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:07:05.452069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.450128 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:05.476473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.476428 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:07:05.498291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.498249 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:05.669066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.657178 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:05.669066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.667226 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:07:05.669066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.667667 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:07:05.765105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.753124 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:07:05.790064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.783969 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Nov 01 16:07:05.843190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.825478 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:07:05.921065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.918168 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:05.930064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:05.924124 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.204116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.203154 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.306107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.305125 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.660789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.657704 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 01 16:07:06.660789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.658561 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:07:06.684573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.668787 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.684573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.669245 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 01 16:07:06.684573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.669928 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:07:06.684573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.670005 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:07:06.936590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.936007 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.951072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:06.937120 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:07:07.076015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:07.070573 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:07:07.176595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:07.176560 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:07:07.381343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:07.380188 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:07.381343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:07.381174 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:07:07.672652 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:07.672145 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:07.712625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:07.682702 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:07.820231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:07.820188 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:08.008118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:08.005753 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:07:08.125373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:08.125341 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:07:08.186062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:08.173152 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 01 16:07:08.483183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:08.482850 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:08.535810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:08.533442 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:07:09.630447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.630366 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz"] Nov 01 16:07:09.649144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.648238 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz"] Nov 01 16:07:09.649144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.648374 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.660195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.660143 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9a482b-openshift-default-dockercfg-bmbzh\"" Nov 01 16:07:09.735310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735260 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsjjc\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-kube-api-access-hsjjc\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.735503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735342 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.735503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735401 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.735653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735607 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.735712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735650 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.735797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735768 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.735858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735832 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.735999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.735930 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.736088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.736050 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.843633 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.843747 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsjjc\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-kube-api-access-hsjjc\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.843772 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.843843 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.843986 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.844008 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.844058 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.844101 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.847073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.844180 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.859065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.855225 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.867152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.860814 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.867152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.861571 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.871059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.870175 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.871202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.871061 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.940256 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.940156 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsjjc\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-kube-api-access-hsjjc\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.946026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.945994 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.949686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.949632 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:09.952301 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:09.950566 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:10.010525 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:10.010490 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:10.613098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:10.613001 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz"] Nov 01 16:07:10.613980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:10.613940 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:10.618159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:10.615266 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:10.618159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:10.615383 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:10.921659 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:07:10.921529 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:07:11.352185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:11.350615 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" event={"ID":"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe","Type":"ContainerStarted","Data":"21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c"} Nov 01 16:07:11.352185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:11.350716 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" event={"ID":"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe","Type":"ContainerStarted","Data":"ab6a8dbd2f5be63a3daef824eb04226133bdc927bf2dc619efdd018feb85d303"} Nov 01 16:07:11.396094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:11.391660 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podStartSLOduration=2.391640789 podStartE2EDuration="2.391640789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:11.385800788 +0000 UTC m=+1171.530255497" watchObservedRunningTime="2026-11-01 16:07:11.391640789 +0000 UTC m=+1171.536095432" Nov 01 16:07:12.014548 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:12.013989 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:12.022405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:12.022152 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:12.376155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:12.375752 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:12.377309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:12.377288 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:07:15.413739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:15.413706 4911 generic.go:356] "Generic (PLEG): container finished" podID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerID="ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45" exitCode=0 Nov 01 16:07:15.414315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:15.413790 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" event={"ID":"a74269e5-3eee-4c31-acf1-a610e61b8ed3","Type":"ContainerDied","Data":"ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45"} Nov 01 16:07:15.414482 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:15.414462 4911 scope.go:122] "RemoveContainer" containerID="ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45" Nov 01 16:07:15.417380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:15.417015 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:15.417380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:15.417147 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:15.417380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:15.417198 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:16.425341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:16.425295 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" event={"ID":"a74269e5-3eee-4c31-acf1-a610e61b8ed3","Type":"ContainerStarted","Data":"b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62"} Nov 01 16:07:16.484593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:16.484551 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:07:16.485772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:16.485736 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 16:07:16.485923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:16.485799 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 16:07:17.482065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:17.479485 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 16:07:17.482065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:17.479558 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 16:07:18.479408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:18.479371 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 16:07:18.479810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:18.479776 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 16:07:19.480532 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:19.480497 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 16:07:19.481137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:19.481100 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 16:07:20.479199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:20.479158 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 16:07:20.479522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:20.479488 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 16:07:21.478805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:21.478766 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 01 16:07:21.479342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:21.478933 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 01 16:07:22.487324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:22.487288 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:07:22.488678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:22.488650 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:07:22.489753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:22.489732 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:07:22.938854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:22.938747 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65"] Nov 01 16:07:22.943907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:22.943879 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:22.956791 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:22.956705 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-734859-serving-cert\"" Nov 01 16:07:22.967145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:22.967112 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65"] Nov 01 16:07:23.108244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.108212 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75m8k\" (UniqueName: \"kubernetes.io/projected/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-kube-api-access-75m8k\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.108463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.108384 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.108463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.108449 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.108631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.108611 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.210386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.210291 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.210551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.210520 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75m8k\" (UniqueName: \"kubernetes.io/projected/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-kube-api-access-75m8k\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.210614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.210601 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.210660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.210650 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.210855 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:07:23.210840 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-serving-cert: secret "maas-api-e2e-shared-b-w3-734859-serving-cert" not found Nov 01 16:07:23.210920 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:07:23.210911 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls podName:de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47 nodeName:}" failed. No retries permitted until 2026-11-01 16:07:23.710891522 +0000 UTC m=+1183.855346156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls") pod "maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" (UID: "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47") : secret "maas-api-e2e-shared-b-w3-734859-serving-cert" not found Nov 01 16:07:23.212519 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.212485 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.214287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.214256 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.221522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.221497 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75m8k\" (UniqueName: \"kubernetes.io/projected/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-kube-api-access-75m8k\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.723803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.722693 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.743058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.741772 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:23.875689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:23.875649 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:24.177431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:24.177379 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65"] Nov 01 16:07:24.178136 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:07:24.177912 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0c13ae_cdf1_4eb3_a843_c07f0c3a7a47.slice/crio-5d5920fce0d28025b0a7c253829a70600a53847f1a3558ac48708242a001eb27 WatchSource:0}: Error finding container 5d5920fce0d28025b0a7c253829a70600a53847f1a3558ac48708242a001eb27: Status 404 returned error can't find the container with id 5d5920fce0d28025b0a7c253829a70600a53847f1a3558ac48708242a001eb27 Nov 01 16:07:24.534392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:24.534352 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" event={"ID":"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47","Type":"ContainerStarted","Data":"5d5920fce0d28025b0a7c253829a70600a53847f1a3558ac48708242a001eb27"} Nov 01 16:07:25.544387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.544350 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" event={"ID":"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47","Type":"ContainerStarted","Data":"4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd"} Nov 01 16:07:25.544873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.544429 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:25.567494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.567428 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" podStartSLOduration=2.97356736 podStartE2EDuration="3.567414638s" totalImagesPullingTime="593.847278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:25.56247947 +0000 UTC m=+1185.706934124" watchObservedRunningTime="2026-11-01 16:07:25.567414638 +0000 UTC m=+1185.711869278" Nov 01 16:07:25.641930 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.641891 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx"] Nov 01 16:07:25.646840 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.646815 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.653270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.652709 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-9a482b-serving-cert\"" Nov 01 16:07:25.658327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.658297 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx"] Nov 01 16:07:25.702768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.702731 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsd9r\" (UniqueName: \"kubernetes.io/projected/ead98060-0354-4a0d-b90c-92ee70db832d-kube-api-access-vsd9r\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.703005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.702778 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.703005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.702935 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.703181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.703050 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead98060-0354-4a0d-b90c-92ee70db832d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.805438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.805335 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsd9r\" (UniqueName: \"kubernetes.io/projected/ead98060-0354-4a0d-b90c-92ee70db832d-kube-api-access-vsd9r\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.805438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.805404 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.805753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.805538 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.805753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.805671 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead98060-0354-4a0d-b90c-92ee70db832d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.805886 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:07:25.805849 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-serving-cert: secret "maas-api-e2e-shared-a-w4-9a482b-serving-cert" not found Nov 01 16:07:25.805960 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:07:25.805922 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls podName:ead98060-0354-4a0d-b90c-92ee70db832d nodeName:}" failed. No retries permitted until 2026-11-01 16:07:26.30590261 +0000 UTC m=+1186.450357229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls") pod "maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" (UID: "ead98060-0354-4a0d-b90c-92ee70db832d") : secret "maas-api-e2e-shared-a-w4-9a482b-serving-cert" not found Nov 01 16:07:25.806908 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.806529 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead98060-0354-4a0d-b90c-92ee70db832d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.810095 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.810066 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:25.823999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:25.823971 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsd9r\" (UniqueName: \"kubernetes.io/projected/ead98060-0354-4a0d-b90c-92ee70db832d-kube-api-access-vsd9r\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:26.314974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:26.314937 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:26.317655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:26.317628 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:26.570250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:26.569337 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:26.892069 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:07:26.892007 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead98060_0354_4a0d_b90c_92ee70db832d.slice/crio-1b3fddaae6b9de9172d1a00db9885cd79a905a066a920b450a133b10de36a170 WatchSource:0}: Error finding container 1b3fddaae6b9de9172d1a00db9885cd79a905a066a920b450a133b10de36a170: Status 404 returned error can't find the container with id 1b3fddaae6b9de9172d1a00db9885cd79a905a066a920b450a133b10de36a170 Nov 01 16:07:26.892860 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:26.892835 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx"] Nov 01 16:07:27.593073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:27.592966 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" event={"ID":"ead98060-0354-4a0d-b90c-92ee70db832d","Type":"ContainerStarted","Data":"2c5cb48a550ce4d9e7a72328476ebaf79af2134766a4b982bcbcb2b0d3627927"} Nov 01 16:07:27.593513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:27.593079 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" event={"ID":"ead98060-0354-4a0d-b90c-92ee70db832d","Type":"ContainerStarted","Data":"1b3fddaae6b9de9172d1a00db9885cd79a905a066a920b450a133b10de36a170"} Nov 01 16:07:27.593513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:27.593111 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:27.613120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:27.613034 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" podStartSLOduration=2.174054372 podStartE2EDuration="2.6130163s" totalImagesPullingTime="438.961928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:27.610845149 +0000 UTC m=+1187.755299792" watchObservedRunningTime="2026-11-01 16:07:27.6130163 +0000 UTC m=+1187.757470942" Nov 01 16:07:31.648444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:31.648411 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:07:33.607661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:33.607628 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:07:40.280375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:40.280334 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:07:40.280820 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:40.280525 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:07:40.871661 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:07:40.871624 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:07:49.754457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:49.754158 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7"] Nov 01 16:07:49.843059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:49.837905 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:49.902889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:49.902851 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-59d3a7-serving-cert\"" Nov 01 16:07:50.385650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.385601 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phgj\" (UniqueName: \"kubernetes.io/projected/505cc58d-c14e-4d4a-93fc-7b81c59405d6-kube-api-access-4phgj\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.386009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.385979 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-maas-api-tls\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.386278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.386257 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505cc58d-c14e-4d4a-93fc-7b81c59405d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.386621 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.386596 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-metrics-certs\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.499486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.499442 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phgj\" (UniqueName: \"kubernetes.io/projected/505cc58d-c14e-4d4a-93fc-7b81c59405d6-kube-api-access-4phgj\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.499682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.499609 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-maas-api-tls\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.499756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.499721 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505cc58d-c14e-4d4a-93fc-7b81c59405d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.500020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.499999 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-metrics-certs\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.504203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.501312 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505cc58d-c14e-4d4a-93fc-7b81c59405d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.540734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.524054 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7"] Nov 01 16:07:50.542065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.542014 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-maas-api-tls\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.545095 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.545071 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-metrics-certs\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.557807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.557728 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phgj\" (UniqueName: \"kubernetes.io/projected/505cc58d-c14e-4d4a-93fc-7b81c59405d6-kube-api-access-4phgj\") pod \"maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:50.803397 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:50.803339 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:51.331072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.276985 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.463352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.457914 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:07:51.463352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.458843 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.560224 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.507165 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:07:51.703949 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.702410 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:07:51.759156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.758594 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:07:51.877064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.858600 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.929061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:51.922665 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.220536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.220494 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.286600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.286501 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 01 16:07:52.348558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.341309 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7"] Nov 01 16:07:52.348558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.341459 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:07:52.348558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.344859 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.390371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.390320 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.395588 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.395553 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.461061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.448917 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.479486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.478632 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.480511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.480484 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:07:52.496834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.486133 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:07:52.496834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.486953 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:07:52.508366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.508326 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Nov 01 16:07:52.514732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.512514 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:07:52.524383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.524357 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:07:52.538663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.538256 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.719202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.719156 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.745180 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.745136 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:07:52.746105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.746065 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 01 16:07:52.746334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.746306 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:07:52.919171 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.919063 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.922732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.921962 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:07:52.951112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.943700 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:07:52.975997 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.954630 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:07:52.975997 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.966851 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:07:52.977387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:52.977353 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" event={"ID":"505cc58d-c14e-4d4a-93fc-7b81c59405d6","Type":"ContainerStarted","Data":"4637bc6e6fda454002305c96c0f99f072061edcffebfc61fc77f51db52c418fd"} Nov 01 16:07:53.180139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:53.176025 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:07:53.359317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:53.359285 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:07:53.987459 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:53.987418 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" event={"ID":"505cc58d-c14e-4d4a-93fc-7b81c59405d6","Type":"ContainerStarted","Data":"3bdd857b863aa67cd8165632343608dd8eff4e7579ac1e622aaec8becd2e7e10"} Nov 01 16:07:53.987901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:53.987716 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:07:54.012230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:07:54.012171 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" podStartSLOduration=3.956132825 podStartE2EDuration="5.012156216s" totalImagesPullingTime="1.056023391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:54.007878261 +0000 UTC m=+1214.152332902" watchObservedRunningTime="2026-11-01 16:07:54.012156216 +0000 UTC m=+1214.156610914" Nov 01 16:08:00.002966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:00.002937 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:08:05.698500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.698459 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9"] Nov 01 16:08:05.703656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.703628 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.712346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.712320 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f47890-openshift-default-dockercfg-qpfmq\"" Nov 01 16:08:05.715241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.715205 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9"] Nov 01 16:08:05.721607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.721583 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6"] Nov 01 16:08:05.728253 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.728228 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.769177 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.769098 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6"] Nov 01 16:08:05.776581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.776542 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs"] Nov 01 16:08:05.782776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.782745 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:05.784225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.784182 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs"] Nov 01 16:08:05.793616 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.793586 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.793723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.793668 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.793767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.793740 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.793816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.793801 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.793865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.793830 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.793964 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.793942 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.794028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.793994 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.794112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794087 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.794153 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794140 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.794257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794241 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.794329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794313 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79q6z\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-kube-api-access-79q6z\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.794391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794374 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.794447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794402 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.794496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794447 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r489f\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-kube-api-access-r489f\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.794688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794667 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.794732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794716 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.794806 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794784 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.794919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.794901 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.812566 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.812537 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-f01de5-serving-cert\"" Nov 01 16:08:05.945651 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.945588 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.947761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.947727 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.948556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.948493 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:05.948804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.948780 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79q6z\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-kube-api-access-79q6z\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.948983 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949012 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949027 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949112 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r489f\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-kube-api-access-r489f\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949309 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949365 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949403 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949498 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.949007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949524 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lstwt\" (UniqueName: \"kubernetes.io/projected/e3b19626-9d05-4f9c-835a-e1103e47e657-kube-api-access-lstwt\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:05.950215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.949928 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.950335 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950311 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:05.950408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950390 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.950464 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950444 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.950520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950476 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.950577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950519 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.950577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950530 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.950577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950566 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.950745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950618 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.950745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950668 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.950745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950697 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b19626-9d05-4f9c-835a-e1103e47e657-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:05.950745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950736 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.950946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.950796 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.951370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.951313 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.952712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.951665 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.952712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.951682 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.952712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.951986 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.952712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.952016 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.952712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.952356 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.954778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.954700 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:05.955155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.955102 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:05.973561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:05.973290 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r489f\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-kube-api-access-r489f\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:06.001699 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.001665 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:06.005800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.005762 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:06.006514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.006462 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:06.048151 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.048111 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:06.049091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.049032 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79q6z\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-kube-api-access-79q6z\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:06.051586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.051557 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:06.052338 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.052307 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b19626-9d05-4f9c-835a-e1103e47e657-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.052441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.052401 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.052704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.052679 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lstwt\" (UniqueName: \"kubernetes.io/projected/e3b19626-9d05-4f9c-835a-e1103e47e657-kube-api-access-lstwt\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.052800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.052737 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.053295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.053217 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b19626-9d05-4f9c-835a-e1103e47e657-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.061205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.057711 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.061909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.061830 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.071631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.071557 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lstwt\" (UniqueName: \"kubernetes.io/projected/e3b19626-9d05-4f9c-835a-e1103e47e657-kube-api-access-lstwt\") pod \"maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.091121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.088016 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:06.104821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.099267 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:06.394104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.394062 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.394410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.394383 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.394581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.394559 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.425971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.425920 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9"] Nov 01 16:08:06.520286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.511994 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6"] Nov 01 16:08:06.640295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.633956 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs"] Nov 01 16:08:06.644535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.644493 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.644673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.644585 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.644673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:06.644635 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.701864 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:08:06.701829 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b19626_9d05_4f9c_835a_e1103e47e657.slice/crio-5a78f548448ad928a6191e940152e5fc31daad207732d582c1bcbfd8b7099a6e WatchSource:0}: Error finding container 5a78f548448ad928a6191e940152e5fc31daad207732d582c1bcbfd8b7099a6e: Status 404 returned error can't find the container with id 5a78f548448ad928a6191e940152e5fc31daad207732d582c1bcbfd8b7099a6e Nov 01 16:08:07.146167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:07.145676 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" event={"ID":"6eaeb7c7-bba4-4e11-83aa-25832f3f5797","Type":"ContainerStarted","Data":"62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65"} Nov 01 16:08:07.146167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:07.145733 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" event={"ID":"6eaeb7c7-bba4-4e11-83aa-25832f3f5797","Type":"ContainerStarted","Data":"818cf3a35990dab327d521c42ee19182b0d8f2b7b8af21fdb598a6b146aeef47"} Nov 01 16:08:07.147991 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:07.147957 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" event={"ID":"e3b19626-9d05-4f9c-835a-e1103e47e657","Type":"ContainerStarted","Data":"5a78f548448ad928a6191e940152e5fc31daad207732d582c1bcbfd8b7099a6e"} Nov 01 16:08:07.150377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:07.150350 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" event={"ID":"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1","Type":"ContainerStarted","Data":"6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d"} Nov 01 16:08:07.150490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:07.150404 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" event={"ID":"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1","Type":"ContainerStarted","Data":"fded38dcbc8eb84ed307373291f19b2f72ca76529c3e2c35cbc04624dad34d5b"} Nov 01 16:08:07.168769 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:07.168707 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podStartSLOduration=2.168689113 podStartE2EDuration="2.168689113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:07.166296766 +0000 UTC m=+1227.310751409" watchObservedRunningTime="2026-11-01 16:08:07.168689113 +0000 UTC m=+1227.313143755" Nov 01 16:08:07.194596 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:07.194532 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podStartSLOduration=2.194513182 podStartE2EDuration="2.194513182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:07.19216497 +0000 UTC m=+1227.336619605" watchObservedRunningTime="2026-11-01 16:08:07.194513182 +0000 UTC m=+1227.338967824" Nov 01 16:08:08.080066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.057460 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:08.082058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.082008 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 16:08:08.082282 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.082256 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 16:08:08.098410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.098363 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:08.101491 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.101459 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 16:08:08.101730 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.101705 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 16:08:08.213186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.211826 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" event={"ID":"e3b19626-9d05-4f9c-835a-e1103e47e657","Type":"ContainerStarted","Data":"aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337"} Nov 01 16:08:08.213186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.212901 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:08.233630 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:08.233575 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" podStartSLOduration=2.70632574 podStartE2EDuration="3.23356127s" totalImagesPullingTime="527.23553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:08.229887691 +0000 UTC m=+1228.374342332" watchObservedRunningTime="2026-11-01 16:08:08.23356127 +0000 UTC m=+1228.378015911" Nov 01 16:08:09.075595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:09.074460 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 16:08:09.075595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:09.074525 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 16:08:09.112628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:09.112229 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 16:08:09.112628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:09.112299 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 16:08:10.084065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:10.062514 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 16:08:10.084065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:10.062585 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 16:08:10.104978 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:10.104919 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 16:08:10.105322 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:10.105237 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 16:08:10.927080 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:08:10.925738 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:08:11.057460 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:11.057416 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 16:08:11.065067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:11.057719 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 16:08:11.134578 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:11.134538 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 16:08:11.134892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:11.134857 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 16:08:12.072065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:12.063448 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 16:08:12.072065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:12.063515 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 16:08:12.108183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:12.108144 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 16:08:12.108457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:12.108427 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 16:08:13.064065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:13.062620 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 01 16:08:13.064065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:13.062692 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 01 16:08:13.102089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:13.098466 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 01 16:08:13.102089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:13.098545 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 01 16:08:14.058487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:14.058454 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:14.058968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:14.058944 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:14.059916 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:14.059891 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:08:14.101821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:14.101786 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:14.102303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:14.102280 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:14.103231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:14.103208 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:08:15.351412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:15.351377 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:08:40.908021 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:08:40.907975 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:08:43.494202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.494091 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b"] Nov 01 16:08:43.501422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.501391 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b"] Nov 01 16:08:43.501608 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.501514 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.505534 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.505502 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5c6b75-openshift-default-dockercfg-j2qvf\"" Nov 01 16:08:43.594013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.593977 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-data\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594087 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-podinfo\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594194 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594225 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jdn\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-kube-api-access-z8jdn\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594253 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-envoy\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594365 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-certs\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594412 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-token\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594461 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istiod-ca-cert\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.594570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.594557 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-credential-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.696482 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.696439 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-data\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.696845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.696819 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-data\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.697017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.696999 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-podinfo\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.697148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.697077 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.697148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.697103 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jdn\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-kube-api-access-z8jdn\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.697148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.697141 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-envoy\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.697317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.697196 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-certs\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.697317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.697258 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-token\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.697612 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.697593 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-certs\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.698927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.698839 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istiod-ca-cert\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.699678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.699185 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.699794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.699743 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istiod-ca-cert\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.699968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.699948 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-credential-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.700396 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.700369 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-credential-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.702610 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.702548 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-envoy\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.708641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.708618 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-podinfo\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.713424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.713376 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jdn\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-kube-api-access-z8jdn\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.732761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.732729 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-token\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:43.822393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:43.822301 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:44.542511 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:08:44.542474 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e3b979_dbc3_4753_b187_46fdbdc84b7d.slice/crio-e823ac3ee7b42f67dd757d2e69cfea57f7fc18a7484f921c030583e6753d8bf4 WatchSource:0}: Error finding container e823ac3ee7b42f67dd757d2e69cfea57f7fc18a7484f921c030583e6753d8bf4: Status 404 returned error can't find the container with id e823ac3ee7b42f67dd757d2e69cfea57f7fc18a7484f921c030583e6753d8bf4 Nov 01 16:08:44.543655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.543377 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b"] Nov 01 16:08:44.548021 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.547980 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:44.548262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.548238 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:44.548464 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.548446 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:44.793961 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.793841 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" event={"ID":"d7e3b979-dbc3-4753-b187-46fdbdc84b7d","Type":"ContainerStarted","Data":"b67f485312fa7d5cbcbe719ca3624a946eed8254ee1cd9c88920eebd5e81c2da"} Nov 01 16:08:44.793961 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.793910 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" event={"ID":"d7e3b979-dbc3-4753-b187-46fdbdc84b7d","Type":"ContainerStarted","Data":"e823ac3ee7b42f67dd757d2e69cfea57f7fc18a7484f921c030583e6753d8bf4"} Nov 01 16:08:44.823253 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.823163 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:44.834757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:44.832966 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podStartSLOduration=1.832944655 podStartE2EDuration="1.832944655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:44.826158813 +0000 UTC m=+1264.970613480" watchObservedRunningTime="2026-11-01 16:08:44.832944655 +0000 UTC m=+1264.977399297" Nov 01 16:08:45.831078 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:45.831030 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:46.820124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:46.820090 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:08:46.820887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:08:46.820867 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:09:10.868395 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:09:10.868353 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:09:20.297202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:20.297145 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:20.354980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:20.354932 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:20.374177 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:20.374128 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:20.533066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:20.528310 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:20.686848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:20.686743 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:20.960159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:20.959203 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:09:20.976206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:20.976155 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:21.050231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:21.050179 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:21.090875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:21.090829 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:21.176305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:21.176205 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:21.180175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:21.180116 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:21.370534 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:21.370488 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:31.214270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.214229 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4"] Nov 01 16:09:31.216515 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.216457 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" podUID="0efa9f9d-1f9f-4612-877e-cf1819dab209" containerName="maas-api" containerID="cri-o://5d7ee57b412404553f3b8826ff12907dacf1243493b62299b9ab135ebc9f14a2" gracePeriod=30 Nov 01 16:09:31.351865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.351821 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7"] Nov 01 16:09:31.352968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.352908 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" podUID="505cc58d-c14e-4d4a-93fc-7b81c59405d6" containerName="maas-api" containerID="cri-o://3bdd857b863aa67cd8165632343608dd8eff4e7579ac1e622aaec8becd2e7e10" gracePeriod=30 Nov 01 16:09:31.832429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.832342 4911 generic.go:356] "Generic (PLEG): container finished" podID="0efa9f9d-1f9f-4612-877e-cf1819dab209" containerID="5d7ee57b412404553f3b8826ff12907dacf1243493b62299b9ab135ebc9f14a2" exitCode=0 Nov 01 16:09:31.832949 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.832920 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" event={"ID":"0efa9f9d-1f9f-4612-877e-cf1819dab209","Type":"ContainerDied","Data":"5d7ee57b412404553f3b8826ff12907dacf1243493b62299b9ab135ebc9f14a2"} Nov 01 16:09:31.852684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.852650 4911 generic.go:356] "Generic (PLEG): container finished" podID="505cc58d-c14e-4d4a-93fc-7b81c59405d6" containerID="3bdd857b863aa67cd8165632343608dd8eff4e7579ac1e622aaec8becd2e7e10" exitCode=0 Nov 01 16:09:31.853147 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:31.853124 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" event={"ID":"505cc58d-c14e-4d4a-93fc-7b81c59405d6","Type":"ContainerDied","Data":"3bdd857b863aa67cd8165632343608dd8eff4e7579ac1e622aaec8becd2e7e10"} Nov 01 16:09:32.469942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.469860 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:09:32.502185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.502149 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:09:32.613539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.613504 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505cc58d-c14e-4d4a-93fc-7b81c59405d6-kube-api-access-4phgj\" (UniqueName: \"kubernetes.io/projected/505cc58d-c14e-4d4a-93fc-7b81c59405d6-kube-api-access-4phgj\") pod \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " Nov 01 16:09:32.613717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.613586 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-maas-api-tls\") pod \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " Nov 01 16:09:32.613717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.613677 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efa9f9d-1f9f-4612-877e-cf1819dab209-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efa9f9d-1f9f-4612-877e-cf1819dab209-openshift-service-ca\") pod \"0efa9f9d-1f9f-4612-877e-cf1819dab209\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " Nov 01 16:09:32.613979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.613917 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-metrics-certs\") pod \"0efa9f9d-1f9f-4612-877e-cf1819dab209\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " Nov 01 16:09:32.614155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.614022 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505cc58d-c14e-4d4a-93fc-7b81c59405d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505cc58d-c14e-4d4a-93fc-7b81c59405d6-openshift-service-ca\") pod \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " Nov 01 16:09:32.614155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.614125 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efa9f9d-1f9f-4612-877e-cf1819dab209-openshift-service-ca" pod "0efa9f9d-1f9f-4612-877e-cf1819dab209" (UID: "0efa9f9d-1f9f-4612-877e-cf1819dab209"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:32.614294 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.614184 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-metrics-certs\") pod \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\" (UID: \"505cc58d-c14e-4d4a-93fc-7b81c59405d6\") " Nov 01 16:09:32.614358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.614320 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-maas-api-tls\") pod \"0efa9f9d-1f9f-4612-877e-cf1819dab209\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " Nov 01 16:09:32.614358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.614345 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505cc58d-c14e-4d4a-93fc-7b81c59405d6-openshift-service-ca" pod "505cc58d-c14e-4d4a-93fc-7b81c59405d6" (UID: "505cc58d-c14e-4d4a-93fc-7b81c59405d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:32.614433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.614360 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0efa9f9d-1f9f-4612-877e-cf1819dab209-kube-api-access-842j6\" (UniqueName: \"kubernetes.io/projected/0efa9f9d-1f9f-4612-877e-cf1819dab209-kube-api-access-842j6\") pod \"0efa9f9d-1f9f-4612-877e-cf1819dab209\" (UID: \"0efa9f9d-1f9f-4612-877e-cf1819dab209\") " Nov 01 16:09:32.615028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.615011 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505cc58d-c14e-4d4a-93fc-7b81c59405d6-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.615128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.615030 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efa9f9d-1f9f-4612-877e-cf1819dab209-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.644719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.644659 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efa9f9d-1f9f-4612-877e-cf1819dab209-kube-api-access-842j6" pod "0efa9f9d-1f9f-4612-877e-cf1819dab209" (UID: "0efa9f9d-1f9f-4612-877e-cf1819dab209"). InnerVolumeSpecName "kube-api-access-842j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:32.645672 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.645618 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-metrics-certs" pod "505cc58d-c14e-4d4a-93fc-7b81c59405d6" (UID: "505cc58d-c14e-4d4a-93fc-7b81c59405d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:09:32.646600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.646568 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505cc58d-c14e-4d4a-93fc-7b81c59405d6-kube-api-access-4phgj" pod "505cc58d-c14e-4d4a-93fc-7b81c59405d6" (UID: "505cc58d-c14e-4d4a-93fc-7b81c59405d6"). InnerVolumeSpecName "kube-api-access-4phgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:32.647887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.647854 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-maas-api-tls" pod "505cc58d-c14e-4d4a-93fc-7b81c59405d6" (UID: "505cc58d-c14e-4d4a-93fc-7b81c59405d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:09:32.648000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.647890 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-metrics-certs" pod "0efa9f9d-1f9f-4612-877e-cf1819dab209" (UID: "0efa9f9d-1f9f-4612-877e-cf1819dab209"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:09:32.648183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.648160 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-maas-api-tls" pod "0efa9f9d-1f9f-4612-877e-cf1819dab209" (UID: "0efa9f9d-1f9f-4612-877e-cf1819dab209"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:09:32.716573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.716531 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.716573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.716567 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.716573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.716581 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0efa9f9d-1f9f-4612-877e-cf1819dab209-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.716825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.716595 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842j6\" (UniqueName: \"kubernetes.io/projected/0efa9f9d-1f9f-4612-877e-cf1819dab209-kube-api-access-842j6\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.716825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.716608 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phgj\" (UniqueName: \"kubernetes.io/projected/505cc58d-c14e-4d4a-93fc-7b81c59405d6-kube-api-access-4phgj\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.716825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.716623 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505cc58d-c14e-4d4a-93fc-7b81c59405d6-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:32.863932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.863891 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" Nov 01 16:09:32.863932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.863913 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7" event={"ID":"505cc58d-c14e-4d4a-93fc-7b81c59405d6","Type":"ContainerDied","Data":"4637bc6e6fda454002305c96c0f99f072061edcffebfc61fc77f51db52c418fd"} Nov 01 16:09:32.864240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.864001 4911 scope.go:122] "RemoveContainer" containerID="3bdd857b863aa67cd8165632343608dd8eff4e7579ac1e622aaec8becd2e7e10" Nov 01 16:09:32.864240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.864209 4911 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:09:32.864365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.864259 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:32.865877 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.865855 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" Nov 01 16:09:32.866020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.865855 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4" event={"ID":"0efa9f9d-1f9f-4612-877e-cf1819dab209","Type":"ContainerDied","Data":"ad1152ae3f83dea78463853f993b3a01ca28bc9ef6131c71e0d95d3df0bfc2a2"} Nov 01 16:09:32.896114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.896082 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7"] Nov 01 16:09:32.905079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.905021 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59d3a7-85667894fc-z2gc7"] Nov 01 16:09:32.913966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.913937 4911 scope.go:122] "RemoveContainer" containerID="5d7ee57b412404553f3b8826ff12907dacf1243493b62299b9ab135ebc9f14a2" Nov 01 16:09:32.918864 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.918838 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4"] Nov 01 16:09:32.926572 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:32.926544 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76063d10-78c6f5c5f9-dxwq4"] Nov 01 16:09:33.979425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:33.979386 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0efa9f9d-1f9f-4612-877e-cf1819dab209" path="/var/lib/kubelet/pods/0efa9f9d-1f9f-4612-877e-cf1819dab209/volumes" Nov 01 16:09:33.980246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:33.980180 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505cc58d-c14e-4d4a-93fc-7b81c59405d6" path="/var/lib/kubelet/pods/505cc58d-c14e-4d4a-93fc-7b81c59405d6/volumes" Nov 01 16:09:34.887900 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:34.887864 4911 generic.go:356] "Generic (PLEG): container finished" podID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerID="b67f485312fa7d5cbcbe719ca3624a946eed8254ee1cd9c88920eebd5e81c2da" exitCode=0 Nov 01 16:09:34.888138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:34.887940 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" event={"ID":"d7e3b979-dbc3-4753-b187-46fdbdc84b7d","Type":"ContainerDied","Data":"b67f485312fa7d5cbcbe719ca3624a946eed8254ee1cd9c88920eebd5e81c2da"} Nov 01 16:09:34.888449 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:34.888433 4911 scope.go:122] "RemoveContainer" containerID="b67f485312fa7d5cbcbe719ca3624a946eed8254ee1cd9c88920eebd5e81c2da" Nov 01 16:09:34.890263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:34.890233 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:34.890367 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:34.890303 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:34.890367 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:34.890350 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:09:35.900861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:35.900829 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" event={"ID":"d7e3b979-dbc3-4753-b187-46fdbdc84b7d","Type":"ContainerStarted","Data":"1676fbdf0924cf44e22a89c6c8cb83d29e0a537f3757069ca1fe64bf17464cb6"} Nov 01 16:09:36.822794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:36.822759 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:09:36.822969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:36.822913 4911 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 16:09:36.822969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:36.822957 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 16:09:37.822686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:37.822648 4911 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 16:09:37.823114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:37.822712 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 16:09:38.827999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:38.824967 4911 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 16:09:38.827999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:38.825059 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 16:09:39.830543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:39.830481 4911 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 16:09:39.831149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:39.831114 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 16:09:40.859059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:40.852256 4911 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 16:09:40.859059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:40.852522 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 16:09:40.913260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:40.913206 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:09:40.994462 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:09:40.994421 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:09:41.825103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:41.825058 4911 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 01 16:09:41.825327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:41.825121 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 01 16:09:42.245957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.245914 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv"] Nov 01 16:09:42.249096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.247309 4911 cpu_manager.go:414] "containerMap: removing container" podUID="0efa9f9d-1f9f-4612-877e-cf1819dab209" containerName="maas-api" Nov 01 16:09:42.249096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.247338 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efa9f9d-1f9f-4612-877e-cf1819dab209" containerName="maas-api" Nov 01 16:09:42.249096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.247758 4911 cpu_manager.go:414] "containerMap: removing container" podUID="505cc58d-c14e-4d4a-93fc-7b81c59405d6" containerName="maas-api" Nov 01 16:09:42.249096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.247791 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505cc58d-c14e-4d4a-93fc-7b81c59405d6" containerName="maas-api" Nov 01 16:09:42.249096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.248015 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efa9f9d-1f9f-4612-877e-cf1819dab209" containerName="maas-api" Nov 01 16:09:42.249096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.248053 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="505cc58d-c14e-4d4a-93fc-7b81c59405d6" containerName="maas-api" Nov 01 16:09:42.255247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.255221 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.259212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.259178 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv"] Nov 01 16:09:42.261556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.261525 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-80c4c8-serving-cert\"" Nov 01 16:09:42.379759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.379724 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-maas-api-tls\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.379932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.379778 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edf87f-3528-4c20-a4b7-9efa25b8041f-openshift-service-ca\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.379932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.379796 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-metrics-certs\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.379932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.379904 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blppw\" (UniqueName: \"kubernetes.io/projected/d3edf87f-3528-4c20-a4b7-9efa25b8041f-kube-api-access-blppw\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.482342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.482300 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-maas-api-tls\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.482626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.482600 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edf87f-3528-4c20-a4b7-9efa25b8041f-openshift-service-ca\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.483713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.483687 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-metrics-certs\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.485360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.485325 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blppw\" (UniqueName: \"kubernetes.io/projected/d3edf87f-3528-4c20-a4b7-9efa25b8041f-kube-api-access-blppw\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.485691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.485583 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edf87f-3528-4c20-a4b7-9efa25b8041f-openshift-service-ca\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.512143 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.507656 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-maas-api-tls\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.512968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.512927 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-metrics-certs\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.513628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.513568 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blppw\" (UniqueName: \"kubernetes.io/projected/d3edf87f-3528-4c20-a4b7-9efa25b8041f-kube-api-access-blppw\") pod \"maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.582895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.582864 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:42.999121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.994364 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv"] Nov 01 16:09:42.999121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.996851 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:09:42.999121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:42.998420 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:09:43.000349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:43.000323 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:09:43.032290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:43.032167 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:09:43.131518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:43.130701 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" event={"ID":"d3edf87f-3528-4c20-a4b7-9efa25b8041f","Type":"ContainerStarted","Data":"0cce9ad353cf9ae3d623588fd3432f91e4101c2263fe6eed23bc34d5acb3811f"} Nov 01 16:09:44.138585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:44.138544 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" event={"ID":"d3edf87f-3528-4c20-a4b7-9efa25b8041f","Type":"ContainerStarted","Data":"d7a3962f3e5e0713bfa2313e672388e0efa7eec4b94fa891cfd66b2f1cb4a323"} Nov 01 16:09:44.138996 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:44.138641 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:44.159494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:44.159446 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" podStartSLOduration=1.69931746 podStartE2EDuration="2.159432354s" totalImagesPullingTime="460.114894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:44.157697798 +0000 UTC m=+1324.302152441" watchObservedRunningTime="2026-11-01 16:09:44.159432354 +0000 UTC m=+1324.303886995" Nov 01 16:09:46.278519 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.278482 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc"] Nov 01 16:09:46.289557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.289514 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.297167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.297131 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc"] Nov 01 16:09:46.299007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.298921 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f5dd93\"" Nov 01 16:09:46.299316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.299294 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f5dd93-dockercfg-qbvvd\"" Nov 01 16:09:46.304009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.303984 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq"] Nov 01 16:09:46.310338 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.310318 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:46.343087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.343030 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq"] Nov 01 16:09:46.357774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.357738 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3097fcd8-308a-4974-86f8-28bf25d8ee17-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.358027 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.358004 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92h2x\" (UniqueName: \"kubernetes.io/projected/3097fcd8-308a-4974-86f8-28bf25d8ee17-kube-api-access-92h2x\") pod \"payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.531426 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.531314 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f02b673-119f-412d-85db-d0bd27362550-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:46.553096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.531811 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8q4\" (UniqueName: \"kubernetes.io/projected/0f02b673-119f-412d-85db-d0bd27362550-kube-api-access-rp8q4\") pod \"payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:46.553096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.531979 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3097fcd8-308a-4974-86f8-28bf25d8ee17-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.553096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.534379 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3097fcd8-308a-4974-86f8-28bf25d8ee17-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.553096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.536262 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92h2x\" (UniqueName: \"kubernetes.io/projected/3097fcd8-308a-4974-86f8-28bf25d8ee17-kube-api-access-92h2x\") pod \"payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.594923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.594864 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92h2x\" (UniqueName: \"kubernetes.io/projected/3097fcd8-308a-4974-86f8-28bf25d8ee17-kube-api-access-92h2x\") pod \"payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.641908 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.641610 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:46.713152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.695022 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8q4\" (UniqueName: \"kubernetes.io/projected/0f02b673-119f-412d-85db-d0bd27362550-kube-api-access-rp8q4\") pod \"payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:46.713152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.695453 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f02b673-119f-412d-85db-d0bd27362550-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:46.713152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.696466 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f02b673-119f-412d-85db-d0bd27362550-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:46.738819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.738788 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8q4\" (UniqueName: \"kubernetes.io/projected/0f02b673-119f-412d-85db-d0bd27362550-kube-api-access-rp8q4\") pod \"payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:46.954781 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:46.954742 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:47.123433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:47.119968 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc"] Nov 01 16:09:47.173523 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:47.173476 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" event={"ID":"3097fcd8-308a-4974-86f8-28bf25d8ee17","Type":"ContainerStarted","Data":"559f7ac0d4c82fda4c7bc279973f2f4be67a1e8d0ab59f2cb33a2af22dfba43e"} Nov 01 16:09:47.317938 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:09:47.317909 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f02b673_119f_412d_85db_d0bd27362550.slice/crio-9db5ac039a7262fbf2d1191c302494115ff43dd85de8e4c2979f20265f1eb09a WatchSource:0}: Error finding container 9db5ac039a7262fbf2d1191c302494115ff43dd85de8e4c2979f20265f1eb09a: Status 404 returned error can't find the container with id 9db5ac039a7262fbf2d1191c302494115ff43dd85de8e4c2979f20265f1eb09a Nov 01 16:09:47.320077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:47.320052 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq"] Nov 01 16:09:48.185229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:48.185188 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" event={"ID":"0f02b673-119f-412d-85db-d0bd27362550","Type":"ContainerStarted","Data":"8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f"} Nov 01 16:09:48.185587 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:48.185255 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" event={"ID":"0f02b673-119f-412d-85db-d0bd27362550","Type":"ContainerStarted","Data":"9db5ac039a7262fbf2d1191c302494115ff43dd85de8e4c2979f20265f1eb09a"} Nov 01 16:09:48.185722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:48.185605 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:48.187813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:48.187786 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" event={"ID":"3097fcd8-308a-4974-86f8-28bf25d8ee17","Type":"ContainerStarted","Data":"7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7"} Nov 01 16:09:48.187953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:48.187858 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:48.205545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:48.205501 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" podStartSLOduration=2.205484355 podStartE2EDuration="2.205484355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:48.201691325 +0000 UTC m=+1328.346145967" watchObservedRunningTime="2026-11-01 16:09:48.205484355 +0000 UTC m=+1328.349938998" Nov 01 16:09:48.224570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:48.224519 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" podStartSLOduration=2.224494678 podStartE2EDuration="2.224494678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:09:48.220191569 +0000 UTC m=+1328.364646209" watchObservedRunningTime="2026-11-01 16:09:48.224494678 +0000 UTC m=+1328.368949335" Nov 01 16:09:50.165254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:50.165216 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:09:59.263063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.259854 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:09:59.296065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.288245 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:09:59.512028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.511981 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps"] Nov 01 16:09:59.512347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.512322 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" containerID="cri-o://6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f" gracePeriod=30 Nov 01 16:09:59.564166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.564130 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst"] Nov 01 16:09:59.591873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.591836 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst"] Nov 01 16:09:59.592087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.591937 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.596168 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.595884 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f5dd93-serving-cert\"" Nov 01 16:09:59.619798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.619768 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn"] Nov 01 16:09:59.621474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.621443 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" containerID="cri-o://ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d" gracePeriod=30 Nov 01 16:09:59.631286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.631254 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4"] Nov 01 16:09:59.631771 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.631630 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" containerID="cri-o://15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb" gracePeriod=30 Nov 01 16:09:59.694023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.693982 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.694306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.694185 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.694306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.694231 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-metrics-certs\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.694481 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.694379 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxf5\" (UniqueName: \"kubernetes.io/projected/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-kube-api-access-znxf5\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.796087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.795939 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.796087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.796065 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.796087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.796098 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-metrics-certs\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.796467 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.796167 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxf5\" (UniqueName: \"kubernetes.io/projected/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-kube-api-access-znxf5\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.796627 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:09:59.796602 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-serving-cert: secret "maas-api-e2e-inf-a-f5dd93-serving-cert" not found Nov 01 16:09:59.796692 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:09:59.796673 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls podName:dc407add-a1cd-4b18-b06c-f5f3c3c62f6a nodeName:}" failed. No retries permitted until 2026-11-01 16:10:00.296655787 +0000 UTC m=+1340.441110407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls") pod "maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" (UID: "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a") : secret "maas-api-e2e-inf-a-f5dd93-serving-cert" not found Nov 01 16:09:59.797504 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.797480 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.800215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.800187 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-metrics-certs\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:09:59.805247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:09:59.805221 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxf5\" (UniqueName: \"kubernetes.io/projected/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-kube-api-access-znxf5\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:10:00.136117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:00.136022 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp"] Nov 01 16:10:00.136315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:00.136286 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" containerID="cri-o://b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf" gracePeriod=30 Nov 01 16:10:00.319065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:00.312342 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:10:00.373063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:00.364721 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:10:00.515890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:00.515849 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:10:00.870691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:00.870605 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst"] Nov 01 16:10:01.418641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:01.418597 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:10:01.419088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:01.418676 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:10:01.541025 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:01.540925 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" event={"ID":"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a","Type":"ContainerStarted","Data":"835b5063fa48b47c0b34d3d910560ca7dd572be0fecda65fe754281da3fb3c72"} Nov 01 16:10:01.541025 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:01.541006 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" event={"ID":"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a","Type":"ContainerStarted","Data":"4ba75d3def95a2404a077c3ceceb41209b22ce5074529667f6bf1adb3c4292ec"} Nov 01 16:10:01.541305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:01.541264 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:10:01.563994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:01.563938 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" podStartSLOduration=2.174862645 podStartE2EDuration="2.563922567s" totalImagesPullingTime="389.059922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:01.560312042 +0000 UTC m=+1341.704766718" watchObservedRunningTime="2026-11-01 16:10:01.563922567 +0000 UTC m=+1341.708377229" Nov 01 16:10:05.003034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.002638 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:10:05.188131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.184736 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:10:05.201073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.200989 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-kube-api-access-8dbdw\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-kube-api-access-8dbdw\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.202991 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.202916 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istiod-ca-cert\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.205432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.205409 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-token\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-token\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.205921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.205797 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-data\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.214884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.214655 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-data" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.215211 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-data\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.215312 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-podinfo\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.215356 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-credential-socket\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.215491 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-socket\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.215555 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-envoy\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.215613 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-credential-socket\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.217361 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-certs\") pod \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\" (UID: \"27ed155f-4b1d-4880-a77c-3dcc0f38466c\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.217435 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-istio-token\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-istio-token\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.217496 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-socket\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.217565 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b15167e-6477-40de-8a0e-f072897a52ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b15167e-6477-40de-8a0e-f072897a52ae-istio-podinfo\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.222152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.218757 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.226904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.225944 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-credential-socket" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.226904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.226185 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-certs" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.226904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.226376 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-credential-socket" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.229352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.229033 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-socket" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.229756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.229532 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-data" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.229972 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.229928 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-socket" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.230252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.230208 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istiod-ca-cert" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.250089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.249969 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b15167e-6477-40de-8a0e-f072897a52ae-istio-podinfo" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.250534 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.250490 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-kube-api-access-8dbdw" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "kube-api-access-8dbdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.257029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.256175 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-envoy" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.257029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.256373 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-token" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.258309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.257421 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-podinfo" pod "27ed155f-4b1d-4880-a77c-3dcc0f38466c" (UID: "27ed155f-4b1d-4880-a77c-3dcc0f38466c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.258309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.257541 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-istio-token" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.274703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.274668 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:10:05.320299 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320259 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97794338-5b3b-447a-9aa3-221d194cdd77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97794338-5b3b-447a-9aa3-221d194cdd77-istio-podinfo\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.320466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320335 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97794338-5b3b-447a-9aa3-221d194cdd77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97794338-5b3b-447a-9aa3-221d194cdd77-istiod-ca-cert\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.320466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320392 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-socket\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.320466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320454 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-certs\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.320594 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320556 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-kube-api-access-8scwl\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-kube-api-access-8scwl\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.320635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320591 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-data\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.320747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320726 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b15167e-6477-40de-8a0e-f072897a52ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b15167e-6477-40de-8a0e-f072897a52ae-istiod-ca-cert\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.320789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320767 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-kube-api-access-8sljv\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-kube-api-access-8sljv\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.320888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.320871 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-istio-token\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-istio-token\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.324281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.323493 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-envoy\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.324281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.323659 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-credential-socket\") pod \"97794338-5b3b-447a-9aa3-221d194cdd77\" (UID: \"97794338-5b3b-447a-9aa3-221d194cdd77\") " Nov 01 16:10:05.324281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.323707 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-certs\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.324281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.323739 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-envoy\") pod \"3b15167e-6477-40de-8a0e-f072897a52ae\" (UID: \"3b15167e-6477-40de-8a0e-f072897a52ae\") " Nov 01 16:10:05.325493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.325305 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-credential-socket" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.327293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.326812 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b15167e-6477-40de-8a0e-f072897a52ae-istiod-ca-cert" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.327293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.326946 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-istio-token" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.327851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.327543 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-socket" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.328786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.328740 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-kube-api-access-8sljv" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "kube-api-access-8sljv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.329008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.328985 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-certs" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.329277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.329251 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-certs" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.329420 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.329254 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97794338-5b3b-447a-9aa3-221d194cdd77-istiod-ca-cert" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.330939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.330797 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-envoy" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.331952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.331925 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-envoy" pod "3b15167e-6477-40de-8a0e-f072897a52ae" (UID: "3b15167e-6477-40de-8a0e-f072897a52ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.333357 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333327 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333495 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333521 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333536 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333551 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b15167e-6477-40de-8a0e-f072897a52ae-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333565 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b15167e-6477-40de-8a0e-f072897a52ae-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333579 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sljv\" (UniqueName: \"kubernetes.io/projected/3b15167e-6477-40de-8a0e-f072897a52ae-kube-api-access-8sljv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333601 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333616 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dbdw\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-kube-api-access-8dbdw\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333630 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333644 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333658 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333673 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333688 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.333698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333704 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333718 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333732 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333746 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b15167e-6477-40de-8a0e-f072897a52ae-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333759 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97794338-5b3b-447a-9aa3-221d194cdd77-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333773 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333786 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333802 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ed155f-4b1d-4880-a77c-3dcc0f38466c-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333816 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.333342 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-kube-api-access-8scwl" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "kube-api-access-8scwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.334398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.334130 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-data" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.338147 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.338118 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97794338-5b3b-447a-9aa3-221d194cdd77-istio-podinfo" pod "97794338-5b3b-447a-9aa3-221d194cdd77" (UID: "97794338-5b3b-447a-9aa3-221d194cdd77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.435582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.435470 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8scwl\" (UniqueName: \"kubernetes.io/projected/97794338-5b3b-447a-9aa3-221d194cdd77-kube-api-access-8scwl\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.435582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.435510 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97794338-5b3b-447a-9aa3-221d194cdd77-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.435582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.435525 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97794338-5b3b-447a-9aa3-221d194cdd77-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.524235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.523881 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:10:05.629464 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.629425 4911 generic.go:356] "Generic (PLEG): container finished" podID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerID="6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f" exitCode=0 Nov 01 16:10:05.629677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.629514 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" Nov 01 16:10:05.629677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.629542 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" event={"ID":"27ed155f-4b1d-4880-a77c-3dcc0f38466c","Type":"ContainerDied","Data":"6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f"} Nov 01 16:10:05.629677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.629636 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps" event={"ID":"27ed155f-4b1d-4880-a77c-3dcc0f38466c","Type":"ContainerDied","Data":"e0fb35eda5c88d1c33610bade17bf08e0e71b81c3978e1123322315aceb0e893"} Nov 01 16:10:05.629677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.629665 4911 scope.go:122] "RemoveContainer" containerID="6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f" Nov 01 16:10:05.632163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.632139 4911 generic.go:356] "Generic (PLEG): container finished" podID="97794338-5b3b-447a-9aa3-221d194cdd77" containerID="ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d" exitCode=0 Nov 01 16:10:05.632281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.632169 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" event={"ID":"97794338-5b3b-447a-9aa3-221d194cdd77","Type":"ContainerDied","Data":"ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d"} Nov 01 16:10:05.632281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.632222 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" event={"ID":"97794338-5b3b-447a-9aa3-221d194cdd77","Type":"ContainerDied","Data":"6286b9a32b16f45b4721a19423f9f58a87ae54dcfd40a1744da927442d632fb3"} Nov 01 16:10:05.632281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.632225 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn" Nov 01 16:10:05.633828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.633804 4911 generic.go:356] "Generic (PLEG): container finished" podID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerID="b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf" exitCode=0 Nov 01 16:10:05.633958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.633833 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" event={"ID":"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d","Type":"ContainerDied","Data":"b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf"} Nov 01 16:10:05.633958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.633851 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" Nov 01 16:10:05.633958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.633881 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp" event={"ID":"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d","Type":"ContainerDied","Data":"ae1c83cb6b95cc70efc84e1386642b9e8251a4fabf1e43697488a93ab45617e8"} Nov 01 16:10:05.635650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.635627 4911 generic.go:356] "Generic (PLEG): container finished" podID="3b15167e-6477-40de-8a0e-f072897a52ae" containerID="15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb" exitCode=0 Nov 01 16:10:05.635783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.635677 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" event={"ID":"3b15167e-6477-40de-8a0e-f072897a52ae","Type":"ContainerDied","Data":"15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb"} Nov 01 16:10:05.635783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.635714 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" event={"ID":"3b15167e-6477-40de-8a0e-f072897a52ae","Type":"ContainerDied","Data":"58153db2f4e80769d03950c635afd82a5116134906ed757c1fbeae36ba047b6d"} Nov 01 16:10:05.635783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.635750 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4" Nov 01 16:10:05.647338 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647304 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istiod-ca-cert\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.647478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647385 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-data\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.647478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647434 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-certs\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.647599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647508 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-envoy\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.647599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-socket\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.647726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647633 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-token\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-token\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.647781 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647759 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-kube-api-access-h8ngq\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-kube-api-access-h8ngq\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.647850 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.647833 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-credential-socket\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.648086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.648060 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-podinfo\") pod \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\" (UID: \"7ae43db7-d6c2-40f7-ad3a-a871b1703e8d\") " Nov 01 16:10:05.648543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.648518 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-certs" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.648818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.648796 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.649168 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.649139 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-credential-socket" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.649317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.649297 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-data" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.649700 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.649680 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istiod-ca-cert" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.649952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.649927 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-socket" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.651002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.650972 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-envoy" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.651966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.651946 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-kube-api-access-h8ngq" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "kube-api-access-h8ngq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.652256 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.652232 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-podinfo" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.653186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.653169 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-token" pod "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" (UID: "7ae43db7-d6c2-40f7-ad3a-a871b1703e8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.715818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.715730 4911 scope.go:122] "RemoveContainer" containerID="6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f" Nov 01 16:10:05.716147 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:10:05.716122 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f\": container with ID starting with 6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f not found: ID does not exist" containerID="6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f" Nov 01 16:10:05.716245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.716156 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f"} err="failed to get container status \"6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f\": rpc error: code = NotFound desc = could not find container \"6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f\": container with ID starting with 6f63c1baafb05887ac32666192f81f3799c8563a580b92906a300c0d91b6b56f not found: ID does not exist" Nov 01 16:10:05.716245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.716176 4911 scope.go:122] "RemoveContainer" containerID="ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d" Nov 01 16:10:05.729898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.729762 4911 scope.go:122] "RemoveContainer" containerID="ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d" Nov 01 16:10:05.730241 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:10:05.730188 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d\": container with ID starting with ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d not found: ID does not exist" containerID="ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d" Nov 01 16:10:05.730369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.730234 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d"} err="failed to get container status \"ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d\": rpc error: code = NotFound desc = could not find container \"ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d\": container with ID starting with ec3d0883e9d3cac43a6a723c688c2d0db35aa5411b5aef15e6aca317276ada7d not found: ID does not exist" Nov 01 16:10:05.730369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.730263 4911 scope.go:122] "RemoveContainer" containerID="b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf" Nov 01 16:10:05.749742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.749704 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn"] Nov 01 16:10:05.751005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.750969 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.751005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.751005 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.751265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.751020 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.751265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.751035 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.751265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.751103 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ngq\" (UniqueName: \"kubernetes.io/projected/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-kube-api-access-h8ngq\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.751265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.751119 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.751265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.751134 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.751265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.751147 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.764998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.764621 4911 scope.go:122] "RemoveContainer" containerID="b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf" Nov 01 16:10:05.765164 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:10:05.765119 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf\": container with ID starting with b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf not found: ID does not exist" containerID="b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf" Nov 01 16:10:05.765164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.765149 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf"} err="failed to get container status \"b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf\": rpc error: code = NotFound desc = could not find container \"b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf\": container with ID starting with b2a6e3bc38636db5a09fb4284eae4d3f9bf87d4e01867100528c95112403aeaf not found: ID does not exist" Nov 01 16:10:05.765302 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.765174 4911 scope.go:122] "RemoveContainer" containerID="15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb" Nov 01 16:10:05.774884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.774854 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-jrvqn"] Nov 01 16:10:05.779480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.779453 4911 scope.go:122] "RemoveContainer" containerID="15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb" Nov 01 16:10:05.779868 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:10:05.779839 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb\": container with ID starting with 15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb not found: ID does not exist" containerID="15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb" Nov 01 16:10:05.779982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.779880 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb"} err="failed to get container status \"15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb\": rpc error: code = NotFound desc = could not find container \"15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb\": container with ID starting with 15e7aeab5aadc9e8d98fbe2495748d6336a57b86c9f8301327f616a3426bdecb not found: ID does not exist" Nov 01 16:10:05.810821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.810787 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4"] Nov 01 16:10:05.827019 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.826984 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-67wm4"] Nov 01 16:10:05.871064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.870023 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps"] Nov 01 16:10:05.881416 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.881379 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-qpsps"] Nov 01 16:10:05.982538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.982493 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" path="/var/lib/kubelet/pods/27ed155f-4b1d-4880-a77c-3dcc0f38466c/volumes" Nov 01 16:10:05.983137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.983111 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" path="/var/lib/kubelet/pods/3b15167e-6477-40de-8a0e-f072897a52ae/volumes" Nov 01 16:10:05.983998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:05.983833 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" path="/var/lib/kubelet/pods/97794338-5b3b-447a-9aa3-221d194cdd77/volumes" Nov 01 16:10:06.007704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:06.007610 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp"] Nov 01 16:10:06.021304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:06.021219 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-8c7sp"] Nov 01 16:10:07.593353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:07.593317 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:10:07.978893 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:07.978814 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" path="/var/lib/kubelet/pods/7ae43db7-d6c2-40f7-ad3a-a871b1703e8d/volumes" Nov 01 16:10:10.869504 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:10:10.869466 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:10:40.867363 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:10:40.867325 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:10:50.356707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:50.356591 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b"] Nov 01 16:10:50.357374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:50.357278 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" containerID="cri-o://1676fbdf0924cf44e22a89c6c8cb83d29e0a537f3757069ca1fe64bf17464cb6" gracePeriod=30 Nov 01 16:10:55.632124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.631958 4911 generic.go:356] "Generic (PLEG): container finished" podID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerID="1676fbdf0924cf44e22a89c6c8cb83d29e0a537f3757069ca1fe64bf17464cb6" exitCode=0 Nov 01 16:10:55.633007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.632846 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" event={"ID":"d7e3b979-dbc3-4753-b187-46fdbdc84b7d","Type":"ContainerDied","Data":"1676fbdf0924cf44e22a89c6c8cb83d29e0a537f3757069ca1fe64bf17464cb6"} Nov 01 16:10:55.633007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.632905 4911 scope.go:122] "RemoveContainer" containerID="b67f485312fa7d5cbcbe719ca3624a946eed8254ee1cd9c88920eebd5e81c2da" Nov 01 16:10:55.711413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.711385 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:10:55.804701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804664 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-socket\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.804905 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804767 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-data\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.804905 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804789 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-credential-socket\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.804905 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804809 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-certs\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.804905 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804835 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istiod-ca-cert\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.804905 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804904 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-envoy\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.805270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804961 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-socket" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:55.805270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.804978 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-kube-api-access-z8jdn\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-kube-api-access-z8jdn\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.805270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.805152 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-certs" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:55.805270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.805243 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-data" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:55.805494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.805301 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-podinfo\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.805494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.805367 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-credential-socket" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:55.805494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.805394 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-token\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-token\") pod \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\" (UID: \"d7e3b979-dbc3-4753-b187-46fdbdc84b7d\") " Nov 01 16:10:55.805494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.805406 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istiod-ca-cert" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:55.806366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.806338 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.806480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.806371 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.806480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.806387 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.806480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.806401 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.806480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.806415 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.807737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.807711 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-kube-api-access-z8jdn" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "kube-api-access-z8jdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:55.807877 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.807792 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-envoy" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:55.807982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.807958 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-podinfo" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:55.808184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.808167 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-token" pod "d7e3b979-dbc3-4753-b187-46fdbdc84b7d" (UID: "d7e3b979-dbc3-4753-b187-46fdbdc84b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:55.908059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.907963 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.908059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.907993 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.908059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.908004 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:55.908059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:55.908014 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jdn\" (UniqueName: \"kubernetes.io/projected/d7e3b979-dbc3-4753-b187-46fdbdc84b7d-kube-api-access-z8jdn\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:56.646344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:56.646310 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" event={"ID":"d7e3b979-dbc3-4753-b187-46fdbdc84b7d","Type":"ContainerDied","Data":"e823ac3ee7b42f67dd757d2e69cfea57f7fc18a7484f921c030583e6753d8bf4"} Nov 01 16:10:56.646344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:56.646332 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b" Nov 01 16:10:56.646895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:56.646380 4911 scope.go:122] "RemoveContainer" containerID="1676fbdf0924cf44e22a89c6c8cb83d29e0a537f3757069ca1fe64bf17464cb6" Nov 01 16:10:56.679625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:56.679591 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b"] Nov 01 16:10:56.686842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:56.686812 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-lrc9b"] Nov 01 16:10:57.972218 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:10:57.972183 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" path="/var/lib/kubelet/pods/d7e3b979-dbc3-4753-b187-46fdbdc84b7d/volumes" Nov 01 16:11:01.325780 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.325683 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst"] Nov 01 16:11:01.327546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.327508 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" Nov 01 16:11:01.327717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.327702 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" Nov 01 16:11:01.327816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.327806 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" Nov 01 16:11:01.327891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.327880 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" Nov 01 16:11:01.327982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.327973 4911 cpu_manager.go:414] "containerMap: removing container" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" Nov 01 16:11:01.328083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.328071 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" Nov 01 16:11:01.328181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.328171 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" Nov 01 16:11:01.328266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.328256 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" Nov 01 16:11:01.328432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.328418 4911 cpu_manager.go:414] "containerMap: removing container" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" Nov 01 16:11:01.328537 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.328525 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" Nov 01 16:11:01.328635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.328625 4911 cpu_manager.go:414] "containerMap: removing container" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" Nov 01 16:11:01.328712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.328702 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" Nov 01 16:11:01.329023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.329005 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae43db7-d6c2-40f7-ad3a-a871b1703e8d" containerName="istio-proxy" Nov 01 16:11:01.329164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.329151 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" Nov 01 16:11:01.331683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.329286 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ed155f-4b1d-4880-a77c-3dcc0f38466c" containerName="istio-proxy" Nov 01 16:11:01.331683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.329310 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e3b979-dbc3-4753-b187-46fdbdc84b7d" containerName="istio-proxy" Nov 01 16:11:01.331683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.329320 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b15167e-6477-40de-8a0e-f072897a52ae" containerName="istio-proxy" Nov 01 16:11:01.331683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.329335 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="97794338-5b3b-447a-9aa3-221d194cdd77" containerName="istio-proxy" Nov 01 16:11:01.340677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.340650 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst"] Nov 01 16:11:01.341020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.340991 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.349931 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.349184 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1c65e3-serving-cert\"" Nov 01 16:11:01.437333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.437294 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.438820 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.438779 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.439125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.439099 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvdc\" (UniqueName: \"kubernetes.io/projected/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-kube-api-access-cxvdc\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.439615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.439590 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-metrics-certs\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.542428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.542377 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvdc\" (UniqueName: \"kubernetes.io/projected/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-kube-api-access-cxvdc\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.542682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.542662 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-metrics-certs\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.542804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.542790 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.542901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.542886 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.542951 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:01.542906 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-serving-cert: secret "maas-api-e2e-inf-a-1c65e3-serving-cert" not found Nov 01 16:11:01.542991 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:01.542979 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls podName:25db50cf-d326-4ea3-8bd7-c52af1c7bf86 nodeName:}" failed. No retries permitted until 2026-11-01 16:11:02.042959567 +0000 UTC m=+1402.187414201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls") pod "maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" (UID: "25db50cf-d326-4ea3-8bd7-c52af1c7bf86") : secret "maas-api-e2e-inf-a-1c65e3-serving-cert" not found Nov 01 16:11:01.543644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.543619 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.545474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.545445 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-metrics-certs\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:01.554795 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:01.554748 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvdc\" (UniqueName: \"kubernetes.io/projected/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-kube-api-access-cxvdc\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:02.052531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:02.052491 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:02.053856 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:02.052850 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-serving-cert: secret "maas-api-e2e-inf-a-1c65e3-serving-cert" not found Nov 01 16:11:02.056242 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:02.056225 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls podName:25db50cf-d326-4ea3-8bd7-c52af1c7bf86 nodeName:}" failed. No retries permitted until 2026-11-01 16:11:03.056202324 +0000 UTC m=+1403.200656965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls") pod "maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" (UID: "25db50cf-d326-4ea3-8bd7-c52af1c7bf86") : secret "maas-api-e2e-inf-a-1c65e3-serving-cert" not found Nov 01 16:11:03.093761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:03.093724 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:03.096470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:03.096443 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:03.243349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:03.243316 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:03.550717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:03.550680 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst"] Nov 01 16:11:03.555258 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:11:03.555233 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25db50cf_d326_4ea3_8bd7_c52af1c7bf86.slice/crio-ea81f63ff944a4c66db8be95ff36eabce9f5f0e16719fbba802211aafeea5041 WatchSource:0}: Error finding container ea81f63ff944a4c66db8be95ff36eabce9f5f0e16719fbba802211aafeea5041: Status 404 returned error can't find the container with id ea81f63ff944a4c66db8be95ff36eabce9f5f0e16719fbba802211aafeea5041 Nov 01 16:11:03.795686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:03.786142 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" event={"ID":"25db50cf-d326-4ea3-8bd7-c52af1c7bf86","Type":"ContainerStarted","Data":"ea81f63ff944a4c66db8be95ff36eabce9f5f0e16719fbba802211aafeea5041"} Nov 01 16:11:04.797021 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:04.796983 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" event={"ID":"25db50cf-d326-4ea3-8bd7-c52af1c7bf86","Type":"ContainerStarted","Data":"9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0"} Nov 01 16:11:04.797513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:04.797095 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:04.820508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:04.820454 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" podStartSLOduration=3.369072504 podStartE2EDuration="3.82043739s" totalImagesPullingTime="451.364886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:04.81451576 +0000 UTC m=+1404.958970400" watchObservedRunningTime="2026-11-01 16:11:04.82043739 +0000 UTC m=+1404.964892088" Nov 01 16:11:10.812410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:10.812377 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:11:10.873011 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:10.872968 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:11:14.183007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.182708 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq"] Nov 01 16:11:14.188775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.188749 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.194960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.194928 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-76063d10-openshift-default-dockercfg-8n7f7\"" Nov 01 16:11:14.201946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.201925 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj"] Nov 01 16:11:14.219295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.219262 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq"] Nov 01 16:11:14.219649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.219612 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.244200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.244165 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj"] Nov 01 16:11:14.318189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318155 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318226 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318256 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.318358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318301 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.318470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318377 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.318470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318442 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.318470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318468 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318558 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318584 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e07fa3-6136-447f-a86a-a6224b218159-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.318669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318653 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzl7\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-kube-api-access-wkzl7\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.318710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318697 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljmf\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-kube-api-access-bljmf\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318718 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.318872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318853 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318908 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318886 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318941 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318918 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318978 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318953 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.318978 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.318973 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e07fa3-6136-447f-a86a-a6224b218159-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.319091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.319078 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.421309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421258 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e07fa3-6136-447f-a86a-a6224b218159-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.421486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421353 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzl7\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-kube-api-access-wkzl7\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.421486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421416 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljmf\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-kube-api-access-bljmf\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.421486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421441 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.421676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421539 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.421676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421571 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.421676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421606 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.421676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421634 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.421676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421663 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e07fa3-6136-447f-a86a-a6224b218159-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.421942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421700 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.421942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421880 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.422145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421952 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.422145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.421980 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.422145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.422004 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.422145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.422118 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.422394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.422161 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.422471 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.422453 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.422594 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.422577 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.425197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.423846 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.425197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.423987 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.425197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.424021 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.425197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.424340 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.425197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.424395 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.425563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.425197 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e07fa3-6136-447f-a86a-a6224b218159-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.425563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.425208 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.425563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.425340 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.425731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.425587 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.425857 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.425835 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.426345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.426313 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e07fa3-6136-447f-a86a-a6224b218159-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.428180 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.428136 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.431563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.431541 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.431660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.431598 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.431751 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.431728 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.432190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.432160 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzl7\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-kube-api-access-wkzl7\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.432420 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.432375 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.432821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.432801 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljmf\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-kube-api-access-bljmf\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.523474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.523429 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:14.545316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.545283 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:14.826686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.826658 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq"] Nov 01 16:11:14.828188 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:11:14.828155 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e07fa3_6136_447f_a86a_a6224b218159.slice/crio-745b5ff5962cc43bd4b5ff034c4e288e3a6963ce13662a0ffb131d0a5919d0fe WatchSource:0}: Error finding container 745b5ff5962cc43bd4b5ff034c4e288e3a6963ce13662a0ffb131d0a5919d0fe: Status 404 returned error can't find the container with id 745b5ff5962cc43bd4b5ff034c4e288e3a6963ce13662a0ffb131d0a5919d0fe Nov 01 16:11:14.831289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.831258 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.831388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.831328 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.831388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.831358 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.854281 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:11:14.854240 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fef4e_4590_4eeb_8d5c_04ce7bc1bf39.slice/crio-ee09c4a28675d3d48292a62aa54bc776a6530180ab2a3ef00b512251e225d06d WatchSource:0}: Error finding container ee09c4a28675d3d48292a62aa54bc776a6530180ab2a3ef00b512251e225d06d: Status 404 returned error can't find the container with id ee09c4a28675d3d48292a62aa54bc776a6530180ab2a3ef00b512251e225d06d Nov 01 16:11:14.855235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.855199 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj"] Nov 01 16:11:14.857849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.857812 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.857963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.857881 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.857963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.857911 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.896223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.896197 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" event={"ID":"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39","Type":"ContainerStarted","Data":"ee09c4a28675d3d48292a62aa54bc776a6530180ab2a3ef00b512251e225d06d"} Nov 01 16:11:14.897870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:14.897841 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" event={"ID":"b4e07fa3-6136-447f-a86a-a6224b218159","Type":"ContainerStarted","Data":"745b5ff5962cc43bd4b5ff034c4e288e3a6963ce13662a0ffb131d0a5919d0fe"} Nov 01 16:11:15.916229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:15.916177 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" event={"ID":"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39","Type":"ContainerStarted","Data":"d1624fd5d103228dd387fc9b94f60fd321d7674b86e3a972b84278fe005c7c1d"} Nov 01 16:11:15.918219 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:15.918191 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" event={"ID":"b4e07fa3-6136-447f-a86a-a6224b218159","Type":"ContainerStarted","Data":"4d0800f9eda9d53ee8d5aac577f374e03a6e478ab05f6a4e742613adb3247ff4"} Nov 01 16:11:15.942433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:15.942372 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podStartSLOduration=1.942352047 podStartE2EDuration="1.942352047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:15.936703189 +0000 UTC m=+1416.081157831" watchObservedRunningTime="2026-11-01 16:11:15.942352047 +0000 UTC m=+1416.086806689" Nov 01 16:11:16.009136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:16.009066 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podStartSLOduration=2.009030931 podStartE2EDuration="2.009030931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:16.003670692 +0000 UTC m=+1416.148125336" watchObservedRunningTime="2026-11-01 16:11:16.009030931 +0000 UTC m=+1416.153485574" Nov 01 16:11:16.524177 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:16.524102 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:16.526678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:16.526539 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 16:11:16.526678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:16.526599 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 16:11:16.545797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:16.545766 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:16.547297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:16.547272 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 01 16:11:16.547430 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:16.547338 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 01 16:11:17.532825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:17.532773 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 16:11:17.535184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:17.535144 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 16:11:17.585060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:17.572730 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 01 16:11:17.585060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:17.572813 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 01 16:11:18.526143 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:18.526108 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 16:11:18.526427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:18.526396 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 16:11:18.546057 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:18.546002 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 01 16:11:18.546434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:18.546085 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 01 16:11:19.524557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:19.524522 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 16:11:19.524821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:19.524795 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 16:11:19.545775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:19.545737 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 01 16:11:19.545924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:19.545810 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 01 16:11:20.523921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:20.523889 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 16:11:20.524693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:20.524638 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 16:11:20.546709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:20.546439 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 01 16:11:20.546709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:20.546479 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 01 16:11:21.523844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:21.523803 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 01 16:11:21.524181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:21.523885 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 01 16:11:21.545712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:21.545671 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 01 16:11:21.545899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:21.545740 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 01 16:11:22.171566 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.171515 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh"] Nov 01 16:11:22.183836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.179891 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.209558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.206661 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh"] Nov 01 16:11:22.211819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.210655 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-375350\"/\"test-model-375350-kserve-self-signed-certs\"" Nov 01 16:11:22.211819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.210780 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-375350\"/\"kube-root-ca.crt\"" Nov 01 16:11:22.211819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.210883 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-375350\"/\"openshift-service-ca.crt\"" Nov 01 16:11:22.211819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.210986 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-375350\"/\"default-dockercfg-pwvtx\"" Nov 01 16:11:22.273547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.273500 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-model-cache\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.273547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.273548 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-tmp-dir\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.273810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.273582 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-dshm\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.273810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.273672 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7ws\" (UniqueName: \"kubernetes.io/projected/c34c58b2-eb0e-4628-ba46-98207d6443ee-kube-api-access-ht7ws\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.273927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.273831 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34c58b2-eb0e-4628-ba46-98207d6443ee-tls-certs\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.273927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.273891 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-home\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.384473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.384432 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-model-cache\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.385165 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.385147 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-tmp-dir\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.385593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.385576 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-dshm\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.385722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.385708 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7ws\" (UniqueName: \"kubernetes.io/projected/c34c58b2-eb0e-4628-ba46-98207d6443ee-kube-api-access-ht7ws\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.386216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.385085 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-model-cache\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.386320 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.385506 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-tmp-dir\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.386502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.386474 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34c58b2-eb0e-4628-ba46-98207d6443ee-tls-certs\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.386607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.386542 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-home\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.387027 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.387006 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-home\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.388606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.388581 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-dshm\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.408287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.408253 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34c58b2-eb0e-4628-ba46-98207d6443ee-tls-certs\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.416645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.416309 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7ws\" (UniqueName: \"kubernetes.io/projected/c34c58b2-eb0e-4628-ba46-98207d6443ee-kube-api-access-ht7ws\") pod \"test-model-375350-kserve-5ffdd99597-s6bmh\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.532390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.527392 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:22.546422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.546394 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:22.548737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.548713 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:22.554450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.552211 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:22.573082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.572396 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:22.574403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.573477 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:22.575238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.575192 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:22.978677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:22.978645 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh"] Nov 01 16:11:23.072145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:23.072021 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" event={"ID":"c34c58b2-eb0e-4628-ba46-98207d6443ee","Type":"ContainerStarted","Data":"1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570"} Nov 01 16:11:23.072145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:23.072118 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" event={"ID":"c34c58b2-eb0e-4628-ba46-98207d6443ee","Type":"ContainerStarted","Data":"172cc29edff5688e9af3f9580c2068bc28b3e0effcf8b1c68a53d5b9bbebb6c5"} Nov 01 16:11:23.531063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:23.529419 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:29.215739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.215673 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" podStartSLOduration=7.215656046 podStartE2EDuration="7.215656046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:23.100278688 +0000 UTC m=+1423.244733331" watchObservedRunningTime="2026-11-01 16:11:29.215656046 +0000 UTC m=+1429.360110691" Nov 01 16:11:29.219007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.218979 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd"] Nov 01 16:11:29.248536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.240703 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6"] Nov 01 16:11:29.248536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.241021 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.255250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.255218 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.265390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.265358 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd"] Nov 01 16:11:29.278938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.278878 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6"] Nov 01 16:11:29.342827 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.342792 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.342995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.342848 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.343064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.342984 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.343122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343100 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343225 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343300 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.343351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343329 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096e950-ee83-4e12-a24b-58b171829b5e-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343397 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343360 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343435 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096e950-ee83-4e12-a24b-58b171829b5e-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343483 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343538 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.343611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343596 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343657 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343640 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.343808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343789 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.343895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343856 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrr8h\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-kube-api-access-hrr8h\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.343946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343899 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.343946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343929 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z624v\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-kube-api-access-z624v\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.344059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.343956 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.447296 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447239 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.447296 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447301 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096e950-ee83-4e12-a24b-58b171829b5e-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.447582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447332 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.447582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447368 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096e950-ee83-4e12-a24b-58b171829b5e-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.447582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447406 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.447582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447455 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.447582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447488 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.447582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447536 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.447897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447615 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.447897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447674 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrr8h\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-kube-api-access-hrr8h\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.447897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447717 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.447897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447742 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z624v\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-kube-api-access-z624v\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.447897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447768 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.448200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447910 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.448200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.447978 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.448200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.448066 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.448200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.448127 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.448655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.448239 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.449213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.449181 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.449505 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.449480 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.449607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.449536 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.450744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.449784 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.450744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.449973 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.450744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.450234 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.450744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.450698 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.471106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.455386 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.471370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.458206 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.471510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.469310 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.474456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.474429 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.480326 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.474433 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096e950-ee83-4e12-a24b-58b171829b5e-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.480506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.475104 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096e950-ee83-4e12-a24b-58b171829b5e-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.480796 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.480773 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.505391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.505357 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z624v\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-kube-api-access-z624v\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.510066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.506331 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrr8h\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-kube-api-access-hrr8h\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.511237 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.510841 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.512737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.512619 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.595064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.594346 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:29.595064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.594892 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:29.969227 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:11:29.969192 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096e950_ee83_4e12_a24b_58b171829b5e.slice/crio-abb61bd7db22abdd22301b3381770eacf3baa8b40e514b72faa5658a7d17702a WatchSource:0}: Error finding container abb61bd7db22abdd22301b3381770eacf3baa8b40e514b72faa5658a7d17702a: Status 404 returned error can't find the container with id abb61bd7db22abdd22301b3381770eacf3baa8b40e514b72faa5658a7d17702a Nov 01 16:11:29.980012 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.979945 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.980179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.980025 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.980179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.980071 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.980179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.980094 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.980179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.980133 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.980179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.980164 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.983793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.983766 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6"] Nov 01 16:11:29.983915 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:29.983830 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd"] Nov 01 16:11:30.156059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.155494 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" event={"ID":"b096e950-ee83-4e12-a24b-58b171829b5e","Type":"ContainerStarted","Data":"455723046d38168d3026bab06c373015f8dbd3504c384c1c15b40f13e7a9215b"} Nov 01 16:11:30.156059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.155566 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" event={"ID":"b096e950-ee83-4e12-a24b-58b171829b5e","Type":"ContainerStarted","Data":"abb61bd7db22abdd22301b3381770eacf3baa8b40e514b72faa5658a7d17702a"} Nov 01 16:11:30.164061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.160409 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" event={"ID":"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483","Type":"ContainerStarted","Data":"03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053"} Nov 01 16:11:30.164061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.160470 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" event={"ID":"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483","Type":"ContainerStarted","Data":"bc43a565fbdac24d774ff8a84ba0a8282ee6afe652cbd2ab096405fee3caf9ae"} Nov 01 16:11:30.185337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.185243 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podStartSLOduration=1.185227472 podStartE2EDuration="1.185227472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:30.17753881 +0000 UTC m=+1430.321993456" watchObservedRunningTime="2026-11-01 16:11:30.185227472 +0000 UTC m=+1430.329682112" Nov 01 16:11:30.201452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.201387 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podStartSLOduration=1.2013675400000001 podStartE2EDuration="1.20136754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:30.196215369 +0000 UTC m=+1430.340670012" watchObservedRunningTime="2026-11-01 16:11:30.20136754 +0000 UTC m=+1430.345822182" Nov 01 16:11:30.594786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.594738 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:30.595817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:30.595797 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:31.600146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:31.600104 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:31.600748 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:31.600714 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 16:11:31.617308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:31.617270 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:31.617572 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:31.617546 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 01 16:11:32.528269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:32.528237 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:32.542309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:32.542283 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:32.609603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:32.609563 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:32.610202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:32.610169 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 16:11:32.610341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:32.609790 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:32.610454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:32.610437 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 01 16:11:33.610061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:33.609796 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:33.610061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:33.609858 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 01 16:11:33.610638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:33.610528 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:33.610638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:33.610570 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 16:11:33.630830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:33.629761 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:11:34.596152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:34.595913 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:34.596152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:34.595987 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 16:11:34.596447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:34.596225 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:34.596447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:34.596267 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 01 16:11:35.602066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:35.600367 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:35.602066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:35.600431 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 16:11:35.602066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:35.601274 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:35.602066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:35.601317 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 01 16:11:36.598493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:36.598453 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 01 16:11:36.601206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:36.601172 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 01 16:11:36.601755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:36.599918 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 01 16:11:36.601910 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:36.601889 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 01 16:11:37.274067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.272636 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj"] Nov 01 16:11:37.274067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.272892 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" containerID="cri-o://d1624fd5d103228dd387fc9b94f60fd321d7674b86e3a972b84278fe005c7c1d" gracePeriod=30 Nov 01 16:11:37.291718 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.283128 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd"] Nov 01 16:11:37.291718 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.283359 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" containerID="cri-o://03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053" gracePeriod=30 Nov 01 16:11:37.291924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.291762 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6"] Nov 01 16:11:37.292067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.291980 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" containerID="cri-o://455723046d38168d3026bab06c373015f8dbd3504c384c1c15b40f13e7a9215b" gracePeriod=30 Nov 01 16:11:37.372425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.372387 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq"] Nov 01 16:11:37.373097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.372998 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" containerID="cri-o://4d0800f9eda9d53ee8d5aac577f374e03a6e478ab05f6a4e742613adb3247ff4" gracePeriod=30 Nov 01 16:11:37.597789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.594199 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:11:37.599605 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.599571 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:11:37.600519 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.599067 4911 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 01 16:11:37.600740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:37.600693 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:11:40.870140 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:40.870091 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:11:42.267766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.266310 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv"] Nov 01 16:11:42.267766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.266596 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" podUID="d3edf87f-3528-4c20-a4b7-9efa25b8041f" containerName="maas-api" containerID="cri-o://d7a3962f3e5e0713bfa2313e672388e0efa7eec4b94fa891cfd66b2f1cb4a323" gracePeriod=30 Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.989296 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.993166 4911 generic.go:356] "Generic (PLEG): container finished" podID="d3edf87f-3528-4c20-a4b7-9efa25b8041f" containerID="d7a3962f3e5e0713bfa2313e672388e0efa7eec4b94fa891cfd66b2f1cb4a323" exitCode=0 Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.993346 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" event={"ID":"d3edf87f-3528-4c20-a4b7-9efa25b8041f","Type":"ContainerDied","Data":"d7a3962f3e5e0713bfa2313e672388e0efa7eec4b94fa891cfd66b2f1cb4a323"} Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.993423 4911 scope.go:122] "RemoveContainer" containerID="d7a3962f3e5e0713bfa2313e672388e0efa7eec4b94fa891cfd66b2f1cb4a323" Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.998243 4911 generic.go:356] "Generic (PLEG): container finished" podID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerID="d1624fd5d103228dd387fc9b94f60fd321d7674b86e3a972b84278fe005c7c1d" exitCode=0 Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:42.998425 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" event={"ID":"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39","Type":"ContainerDied","Data":"d1624fd5d103228dd387fc9b94f60fd321d7674b86e3a972b84278fe005c7c1d"} Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.001521 4911 generic.go:356] "Generic (PLEG): container finished" podID="b096e950-ee83-4e12-a24b-58b171829b5e" containerID="455723046d38168d3026bab06c373015f8dbd3504c384c1c15b40f13e7a9215b" exitCode=0 Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.001680 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" event={"ID":"b096e950-ee83-4e12-a24b-58b171829b5e","Type":"ContainerDied","Data":"455723046d38168d3026bab06c373015f8dbd3504c384c1c15b40f13e7a9215b"} Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.004066 4911 generic.go:356] "Generic (PLEG): container finished" podID="b4e07fa3-6136-447f-a86a-a6224b218159" containerID="4d0800f9eda9d53ee8d5aac577f374e03a6e478ab05f6a4e742613adb3247ff4" exitCode=0 Nov 01 16:11:43.008122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.004166 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" event={"ID":"b4e07fa3-6136-447f-a86a-a6224b218159","Type":"ContainerDied","Data":"4d0800f9eda9d53ee8d5aac577f374e03a6e478ab05f6a4e742613adb3247ff4"} Nov 01 16:11:43.148786 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:43.148720 4911 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096e950_ee83_4e12_a24b_58b171829b5e.slice/crio-abb61bd7db22abdd22301b3381770eacf3baa8b40e514b72faa5658a7d17702a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3edf87f_3528_4c20_a4b7_9efa25b8041f.slice/crio-conmon-d7a3962f3e5e0713bfa2313e672388e0efa7eec4b94fa891cfd66b2f1cb4a323.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e07fa3_6136_447f_a86a_a6224b218159.slice/crio-4d0800f9eda9d53ee8d5aac577f374e03a6e478ab05f6a4e742613adb3247ff4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096e950_ee83_4e12_a24b_58b171829b5e.slice/crio-conmon-455723046d38168d3026bab06c373015f8dbd3504c384c1c15b40f13e7a9215b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096e950_ee83_4e12_a24b_58b171829b5e.slice/crio-455723046d38168d3026bab06c373015f8dbd3504c384c1c15b40f13e7a9215b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fef4e_4590_4eeb_8d5c_04ce7bc1bf39.slice/crio-conmon-d1624fd5d103228dd387fc9b94f60fd321d7674b86e3a972b84278fe005c7c1d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e07fa3_6136_447f_a86a_a6224b218159.slice/crio-conmon-4d0800f9eda9d53ee8d5aac577f374e03a6e478ab05f6a4e742613adb3247ff4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faa1eb5_ccb0_4e3c_80de_6f010e9d5483.slice/crio-03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faa1eb5_ccb0_4e3c_80de_6f010e9d5483.slice/crio-conmon-03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:11:43.150469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.150406 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3edf87f-3528-4c20-a4b7-9efa25b8041f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edf87f-3528-4c20-a4b7-9efa25b8041f-openshift-service-ca\") pod \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " Nov 01 16:11:43.150745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.150534 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-maas-api-tls\") pod \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " Nov 01 16:11:43.150745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.150734 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-metrics-certs\") pod \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " Nov 01 16:11:43.150899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.150806 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3edf87f-3528-4c20-a4b7-9efa25b8041f-kube-api-access-blppw\" (UniqueName: \"kubernetes.io/projected/d3edf87f-3528-4c20-a4b7-9efa25b8041f-kube-api-access-blppw\") pod \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\" (UID: \"d3edf87f-3528-4c20-a4b7-9efa25b8041f\") " Nov 01 16:11:43.152800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.152679 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3edf87f-3528-4c20-a4b7-9efa25b8041f-openshift-service-ca" pod "d3edf87f-3528-4c20-a4b7-9efa25b8041f" (UID: "d3edf87f-3528-4c20-a4b7-9efa25b8041f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.161204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.159155 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-metrics-certs" pod "d3edf87f-3528-4c20-a4b7-9efa25b8041f" (UID: "d3edf87f-3528-4c20-a4b7-9efa25b8041f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:43.161204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.159459 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-maas-api-tls" pod "d3edf87f-3528-4c20-a4b7-9efa25b8041f" (UID: "d3edf87f-3528-4c20-a4b7-9efa25b8041f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:43.168031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.161840 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3edf87f-3528-4c20-a4b7-9efa25b8041f-kube-api-access-blppw" pod "d3edf87f-3528-4c20-a4b7-9efa25b8041f" (UID: "d3edf87f-3528-4c20-a4b7-9efa25b8041f"). InnerVolumeSpecName "kube-api-access-blppw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.258842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.258805 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.259031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.258846 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blppw\" (UniqueName: \"kubernetes.io/projected/d3edf87f-3528-4c20-a4b7-9efa25b8041f-kube-api-access-blppw\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.259031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.258865 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3edf87f-3528-4c20-a4b7-9efa25b8041f-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.259031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.258880 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3edf87f-3528-4c20-a4b7-9efa25b8041f-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.312215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.301803 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:43.441106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.441072 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:43.450570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.450452 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:43.470907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.470836 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-kube-api-access-z624v\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-kube-api-access-z624v\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.471112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.471081 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b096e950-ee83-4e12-a24b-58b171829b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096e950-ee83-4e12-a24b-58b171829b5e-istiod-ca-cert\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.471204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.471158 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b096e950-ee83-4e12-a24b-58b171829b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096e950-ee83-4e12-a24b-58b171829b5e-istio-podinfo\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.471268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.471244 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-data\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.471325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.471274 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-credential-socket\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.471382 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.471330 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-certs\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.471382 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.471360 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-socket\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.472439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.471821 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-credential-socket" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.472439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.472065 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-certs" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.472439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.472185 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-socket" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.472439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.472281 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-data" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.472439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.472413 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b096e950-ee83-4e12-a24b-58b171829b5e-istiod-ca-cert" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.473514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.473489 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-istio-token\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.473639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.473546 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-envoy\") pod \"b096e950-ee83-4e12-a24b-58b171829b5e\" (UID: \"b096e950-ee83-4e12-a24b-58b171829b5e\") " Nov 01 16:11:43.474462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.474438 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b096e950-ee83-4e12-a24b-58b171829b5e-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.474574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.474465 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.474574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.474481 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.474574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.474495 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.474574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.474508 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.476783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.476746 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-kube-api-access-z624v" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "kube-api-access-z624v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.479692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.479653 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-envoy" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.479949 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.479923 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-istio-token" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.481400 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.481371 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b096e950-ee83-4e12-a24b-58b171829b5e-istio-podinfo" pod "b096e950-ee83-4e12-a24b-58b171829b5e" (UID: "b096e950-ee83-4e12-a24b-58b171829b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:43.519429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.519403 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:43.576120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.575939 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-data\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.576120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576004 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-envoy\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.576120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576119 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-credential-socket\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.576434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576161 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-socket\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.576434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576204 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-token\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-token\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.576434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576302 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-data\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.576434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576309 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-data" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.576434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576381 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-envoy\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.576434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576413 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-kube-api-access-wkzl7\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-kube-api-access-wkzl7\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576458 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-certs\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576514 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-socket\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576571 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-kube-api-access-bljmf\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-kube-api-access-bljmf\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576598 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-credential-socket\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576620 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-credential-socket" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576722 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istiod-ca-cert\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576753 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-podinfo\") pod \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\" (UID: \"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576841 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e07fa3-6136-447f-a86a-a6224b218159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e07fa3-6136-447f-a86a-a6224b218159-istiod-ca-cert\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.576926 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e07fa3-6136-447f-a86a-a6224b218159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e07fa3-6136-447f-a86a-a6224b218159-istio-podinfo\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577010 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-istio-token\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577059 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-certs\") pod \"b4e07fa3-6136-447f-a86a-a6224b218159\" (UID: \"b4e07fa3-6136-447f-a86a-a6224b218159\") " Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577028 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-socket" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.577368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577358 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-certs" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.578108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577598 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-certs" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.578108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577603 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istiod-ca-cert" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.578108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577811 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-credential-socket" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.578108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.577995 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-socket" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.578661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578580 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578612 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b096e950-ee83-4e12-a24b-58b171829b5e-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578627 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578662 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z624v\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-kube-api-access-z624v\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578679 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578693 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578706 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578719 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b096e950-ee83-4e12-a24b-58b171829b5e-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578732 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578745 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578757 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.578873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.578771 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b096e950-ee83-4e12-a24b-58b171829b5e-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.579334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.579127 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-envoy" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.579334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.579234 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-envoy" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.579455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.579346 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-data" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.579625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.579598 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e07fa3-6136-447f-a86a-a6224b218159-istiod-ca-cert" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.580846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.580810 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-token" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.581515 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.581487 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e07fa3-6136-447f-a86a-a6224b218159-istio-podinfo" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:43.581650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.581619 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-istio-token" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.582561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.582534 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-kube-api-access-wkzl7" pod "b4e07fa3-6136-447f-a86a-a6224b218159" (UID: "b4e07fa3-6136-447f-a86a-a6224b218159"). InnerVolumeSpecName "kube-api-access-wkzl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.582728 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.582703 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-kube-api-access-bljmf" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "kube-api-access-bljmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.582975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.582949 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-podinfo" pod "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" (UID: "7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:43.679847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.679752 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-socket\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.679885 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-token\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-token\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.679927 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-credential-socket\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.679963 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-podinfo\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680028 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-socket" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.680204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680132 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-envoy\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680194 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-certs\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680222 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-kube-api-access-hrr8h\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-kube-api-access-hrr8h\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680271 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-credential-socket" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.680346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680313 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istiod-ca-cert\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.680499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680460 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-data\") pod \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\" (UID: \"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483\") " Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681428 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680460 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-certs" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681457 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681473 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681483 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzl7\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-kube-api-access-wkzl7\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681493 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljmf\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-kube-api-access-bljmf\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681503 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681514 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e07fa3-6136-447f-a86a-a6224b218159-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681527 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e07fa3-6136-447f-a86a-a6224b218159-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680725 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istiod-ca-cert" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681540 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e07fa3-6136-447f-a86a-a6224b218159-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681554 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681568 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e07fa3-6136-447f-a86a-a6224b218159-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.681577 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.681622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.680934 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-data" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.682766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.682730 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-envoy" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.682910 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.682885 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-token" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.683267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.683246 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-podinfo" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:43.683946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.683926 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-kube-api-access-hrr8h" pod "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" (UID: "4faa1eb5-ccb0-4e3c-80de-6f010e9d5483"). InnerVolumeSpecName "kube-api-access-hrr8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.783139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.783096 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.783139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.783130 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.783139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.783140 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.783139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.783149 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.783415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.783160 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrr8h\" (UniqueName: \"kubernetes.io/projected/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-kube-api-access-hrr8h\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.783415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.783170 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.783415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:43.783178 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:44.015635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.015604 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" Nov 01 16:11:44.015800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.015637 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq" event={"ID":"b4e07fa3-6136-447f-a86a-a6224b218159","Type":"ContainerDied","Data":"745b5ff5962cc43bd4b5ff034c4e288e3a6963ce13662a0ffb131d0a5919d0fe"} Nov 01 16:11:44.015800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.015716 4911 scope.go:122] "RemoveContainer" containerID="4d0800f9eda9d53ee8d5aac577f374e03a6e478ab05f6a4e742613adb3247ff4" Nov 01 16:11:44.018818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.017803 4911 generic.go:356] "Generic (PLEG): container finished" podID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerID="03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053" exitCode=0 Nov 01 16:11:44.018818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.018070 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" Nov 01 16:11:44.018818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.018099 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" event={"ID":"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483","Type":"ContainerDied","Data":"03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053"} Nov 01 16:11:44.018818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.018171 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd" event={"ID":"4faa1eb5-ccb0-4e3c-80de-6f010e9d5483","Type":"ContainerDied","Data":"bc43a565fbdac24d774ff8a84ba0a8282ee6afe652cbd2ab096405fee3caf9ae"} Nov 01 16:11:44.020288 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.020216 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" event={"ID":"d3edf87f-3528-4c20-a4b7-9efa25b8041f","Type":"ContainerDied","Data":"0cce9ad353cf9ae3d623588fd3432f91e4101c2263fe6eed23bc34d5acb3811f"} Nov 01 16:11:44.020456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.020291 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv" Nov 01 16:11:44.023368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.022847 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" event={"ID":"7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39","Type":"ContainerDied","Data":"ee09c4a28675d3d48292a62aa54bc776a6530180ab2a3ef00b512251e225d06d"} Nov 01 16:11:44.023368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.022981 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj" Nov 01 16:11:44.027158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.027137 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" event={"ID":"b096e950-ee83-4e12-a24b-58b171829b5e","Type":"ContainerDied","Data":"abb61bd7db22abdd22301b3381770eacf3baa8b40e514b72faa5658a7d17702a"} Nov 01 16:11:44.027158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.027155 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6" Nov 01 16:11:44.030761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.030740 4911 scope.go:122] "RemoveContainer" containerID="03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053" Nov 01 16:11:44.041398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.041376 4911 scope.go:122] "RemoveContainer" containerID="03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053" Nov 01 16:11:44.041728 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:11:44.041703 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053\": container with ID starting with 03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053 not found: ID does not exist" containerID="03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053" Nov 01 16:11:44.041811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.041738 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053"} err="failed to get container status \"03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053\": rpc error: code = NotFound desc = could not find container \"03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053\": container with ID starting with 03d92e26ab5074659ddedf98cfa73247d9511a32f6a26f776a8ab57ac1bc4053 not found: ID does not exist" Nov 01 16:11:44.041811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.041763 4911 scope.go:122] "RemoveContainer" containerID="d1624fd5d103228dd387fc9b94f60fd321d7674b86e3a972b84278fe005c7c1d" Nov 01 16:11:44.054029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.053984 4911 scope.go:122] "RemoveContainer" containerID="455723046d38168d3026bab06c373015f8dbd3504c384c1c15b40f13e7a9215b" Nov 01 16:11:44.070158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.070127 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq"] Nov 01 16:11:44.075409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.075384 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-lvkgq"] Nov 01 16:11:44.107452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.107409 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv"] Nov 01 16:11:44.112595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.112564 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-80c4c8-56f9997f4-4p7hv"] Nov 01 16:11:44.133791 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.133753 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6"] Nov 01 16:11:44.151316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.151286 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-rtfk6"] Nov 01 16:11:44.197222 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.192476 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd"] Nov 01 16:11:44.214035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.214001 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-8wkjd"] Nov 01 16:11:44.231368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.231331 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj"] Nov 01 16:11:44.239808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:44.239776 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-pzhrj"] Nov 01 16:11:45.972110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:45.972066 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" path="/var/lib/kubelet/pods/4faa1eb5-ccb0-4e3c-80de-6f010e9d5483/volumes" Nov 01 16:11:45.972833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:45.972815 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" path="/var/lib/kubelet/pods/7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39/volumes" Nov 01 16:11:45.973472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:45.973456 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" path="/var/lib/kubelet/pods/b096e950-ee83-4e12-a24b-58b171829b5e/volumes" Nov 01 16:11:45.974497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:45.974478 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" path="/var/lib/kubelet/pods/b4e07fa3-6136-447f-a86a-a6224b218159/volumes" Nov 01 16:11:45.975113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:45.975082 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3edf87f-3528-4c20-a4b7-9efa25b8041f" path="/var/lib/kubelet/pods/d3edf87f-3528-4c20-a4b7-9efa25b8041f/volumes" Nov 01 16:11:49.956147 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.956098 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp"] Nov 01 16:11:49.961793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961757 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" Nov 01 16:11:49.961793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961793 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961838 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961847 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961862 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961870 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961886 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d3edf87f-3528-4c20-a4b7-9efa25b8041f" containerName="maas-api" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961894 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3edf87f-3528-4c20-a4b7-9efa25b8041f" containerName="maas-api" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961907 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" Nov 01 16:11:49.962018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.961916 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" Nov 01 16:11:49.963188 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.963165 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="4faa1eb5-ccb0-4e3c-80de-6f010e9d5483" containerName="istio-proxy" Nov 01 16:11:49.963267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.963191 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b096e950-ee83-4e12-a24b-58b171829b5e" containerName="istio-proxy" Nov 01 16:11:49.963267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.963205 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3edf87f-3528-4c20-a4b7-9efa25b8041f" containerName="maas-api" Nov 01 16:11:49.963267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.963217 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4fef4e-4590-4eeb-8d5c-04ce7bc1bf39" containerName="istio-proxy" Nov 01 16:11:49.963267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.963240 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e07fa3-6136-447f-a86a-a6224b218159" containerName="istio-proxy" Nov 01 16:11:49.981788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.981349 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp"] Nov 01 16:11:49.985479 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:49.985443 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.004742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.004711 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-784444-openshift-default-dockercfg-5bhcd\"" Nov 01 16:11:50.156067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.155599 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gm4\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-kube-api-access-b7gm4\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.156067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.155714 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istiod-ca-cert\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.156067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.155871 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-token\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.156067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.155912 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-envoy\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.170375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.156135 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-data\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.170375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.156229 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-socket\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.170375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.156401 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-certs\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.170375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.156576 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-podinfo\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.170375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.156640 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-credential-socket\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258124 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-envoy\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258261 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-data\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258352 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-socket\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258469 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-certs\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258602 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-podinfo\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258658 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-credential-socket\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258763 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258690 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-socket\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258706 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-data\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258698 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gm4\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-kube-api-access-b7gm4\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258869 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istiod-ca-cert\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.258944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258930 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-token\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.259072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258945 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-certs\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.259072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.258986 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-credential-socket\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.259781 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.259749 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istiod-ca-cert\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.261135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.261104 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-envoy\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.261365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.261346 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-podinfo\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.273216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.273190 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-token\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.273355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.273328 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gm4\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-kube-api-access-b7gm4\") pod \"e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:50.316551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:50.316512 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:51.413809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:51.399138 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp"] Nov 01 16:11:51.438363 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:51.438318 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:51.438628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:51.438606 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:51.445922 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:51.438746 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:52.237753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:52.237660 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" event={"ID":"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2","Type":"ContainerStarted","Data":"2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e"} Nov 01 16:11:52.238003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:52.237785 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" event={"ID":"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2","Type":"ContainerStarted","Data":"f875042947ecbd305dbac13163fde81d5aefa03c6ee7029d00e25873a5e07593"} Nov 01 16:11:52.269745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:52.268525 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podStartSLOduration=3.268506324 podStartE2EDuration="3.268506324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:52.264304009 +0000 UTC m=+1452.408758651" watchObservedRunningTime="2026-11-01 16:11:52.268506324 +0000 UTC m=+1452.412960966" Nov 01 16:11:52.318067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:52.317834 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:52.319745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:52.319701 4911 patch_prober.go:29] interesting pod/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 01 16:11:52.319989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:52.319771 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 01 16:11:53.323478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:53.323444 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:54.025231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.025193 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl"] Nov 01 16:11:54.030449 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.030421 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.041459 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.041424 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2ce08a-serving-cert\"" Nov 01 16:11:54.053223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.053186 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl"] Nov 01 16:11:54.228443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.228400 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.229168 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.229139 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-metrics-certs\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.229481 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.229460 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4020e3b4-ec83-4429-89a8-467c965a5e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.229702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.229683 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qv26\" (UniqueName: \"kubernetes.io/projected/4020e3b4-ec83-4429-89a8-467c965a5e5f-kube-api-access-4qv26\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.332993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.332897 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.333598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.333571 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-metrics-certs\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.333875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.333857 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4020e3b4-ec83-4429-89a8-467c965a5e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.334090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.334071 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qv26\" (UniqueName: \"kubernetes.io/projected/4020e3b4-ec83-4429-89a8-467c965a5e5f-kube-api-access-4qv26\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.343399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.342006 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:54.343800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.343767 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4020e3b4-ec83-4429-89a8-467c965a5e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.353098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.346842 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.353098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.347074 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:11:54.383478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.376484 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-metrics-certs\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.401658 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.397077 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qv26\" (UniqueName: \"kubernetes.io/projected/4020e3b4-ec83-4429-89a8-467c965a5e5f-kube-api-access-4qv26\") pod \"maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:54.670581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:54.670475 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:11:55.169377 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:11:55.169338 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4020e3b4_ec83_4429_89a8_467c965a5e5f.slice/crio-adb13b7e3bcba6ad4ff963e46889609eae2095e33aa8fc4447d99c30e2a14549 WatchSource:0}: Error finding container adb13b7e3bcba6ad4ff963e46889609eae2095e33aa8fc4447d99c30e2a14549: Status 404 returned error can't find the container with id adb13b7e3bcba6ad4ff963e46889609eae2095e33aa8fc4447d99c30e2a14549 Nov 01 16:11:55.282319 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:55.258252 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl"] Nov 01 16:11:55.479457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:55.479351 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" event={"ID":"4020e3b4-ec83-4429-89a8-467c965a5e5f","Type":"ContainerStarted","Data":"adb13b7e3bcba6ad4ff963e46889609eae2095e33aa8fc4447d99c30e2a14549"} Nov 01 16:11:56.573064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:56.560113 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" event={"ID":"4020e3b4-ec83-4429-89a8-467c965a5e5f","Type":"ContainerStarted","Data":"24583d522f8b7251aef40aec5b8fbe974b2cac5e492230e41cdd1de8b92c0b6a"} Nov 01 16:11:56.573064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:11:56.560219 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:12:03.582251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:03.582204 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:12:03.601886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:03.601818 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" podStartSLOduration=9.084577534 podStartE2EDuration="9.601797903s" totalImagesPullingTime="517.220369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:56.600681229 +0000 UTC m=+1456.745135871" watchObservedRunningTime="2026-11-01 16:12:03.601797903 +0000 UTC m=+1463.746252545" Nov 01 16:12:06.010858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.010818 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq"] Nov 01 16:12:06.015899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.015875 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.051564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.051511 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq"] Nov 01 16:12:06.106484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.106451 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-375350-openshift-default-dockercfg-wzh7t\"" Nov 01 16:12:06.116682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.116651 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.116846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.116700 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.116846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.116821 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.116974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.116936 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.117078 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.117028 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.117139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.117108 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.117188 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.117172 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.117236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.117220 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.117351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.117335 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drtbf\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-kube-api-access-drtbf\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.218712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.218671 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.218942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.218734 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.218942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.218799 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.218942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.218870 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.218942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.218936 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.219215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.218978 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.219215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.219066 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.219215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.219112 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.219372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.219232 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drtbf\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-kube-api-access-drtbf\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.219711 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.219657 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.219711 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.219694 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.219913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.219815 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.220034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.220012 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.220146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.220120 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.222088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.222060 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.222320 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.222303 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.227091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.227029 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.227264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.227247 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drtbf\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-kube-api-access-drtbf\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-664vq\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.371473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.371374 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:06.693567 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:12:06.693535 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50de64b_b0d0_4cd5_9b16_7c92d8d37f5a.slice/crio-befc2266915292da75ece468d0deee05ebe1a4b4c46cf8c5fd4015bfee63e099 WatchSource:0}: Error finding container befc2266915292da75ece468d0deee05ebe1a4b4c46cf8c5fd4015bfee63e099: Status 404 returned error can't find the container with id befc2266915292da75ece468d0deee05ebe1a4b4c46cf8c5fd4015bfee63e099 Nov 01 16:12:06.693948 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.693924 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq"] Nov 01 16:12:06.696622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.696584 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:06.696741 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.696665 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:06.696741 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:06.696715 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:07.684965 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:07.684923 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" event={"ID":"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a","Type":"ContainerStarted","Data":"32c88a35dff80ca9776af9bcfc891e313199d75534c025b78b81198f9bd4891a"} Nov 01 16:12:07.685550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:07.684982 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" event={"ID":"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a","Type":"ContainerStarted","Data":"befc2266915292da75ece468d0deee05ebe1a4b4c46cf8c5fd4015bfee63e099"} Nov 01 16:12:07.715008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:07.714941 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podStartSLOduration=2.71492159 podStartE2EDuration="2.71492159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:07.711893634 +0000 UTC m=+1467.856348276" watchObservedRunningTime="2026-11-01 16:12:07.71492159 +0000 UTC m=+1467.859376235" Nov 01 16:12:08.378067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:08.376151 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:08.381717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:08.381686 4911 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-664vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 01 16:12:08.381882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:08.381742 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 01 16:12:09.412064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:09.411904 4911 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-664vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 01 16:12:09.412064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:09.411964 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 01 16:12:10.375508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:10.375466 4911 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-664vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 01 16:12:10.375732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:10.375540 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 01 16:12:10.898773 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:12:10.898683 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:12:11.379065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:11.376501 4911 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-664vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 01 16:12:11.379065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:11.376582 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 01 16:12:12.376297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:12.376260 4911 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-664vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 01 16:12:12.376999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:12.376966 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 01 16:12:13.376406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:13.376331 4911 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-664vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 01 16:12:13.376921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:13.376449 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 01 16:12:14.381803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:14.381759 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:14.382826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:14.382802 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:14.383755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:14.383731 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:12:16.571757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.568095 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw"] Nov 01 16:12:16.605157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.605123 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw"] Nov 01 16:12:16.605374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.605273 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.614090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.613117 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-784444-serving-cert\"" Nov 01 16:12:16.683362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.683311 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.683362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.683362 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-metrics-certs\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.683656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.683630 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c254b441-da43-44c6-9bb7-6fa6775b4514-openshift-service-ca\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.683917 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.683891 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4dj\" (UniqueName: \"kubernetes.io/projected/c254b441-da43-44c6-9bb7-6fa6775b4514-kube-api-access-nv4dj\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.785129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.785087 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c254b441-da43-44c6-9bb7-6fa6775b4514-openshift-service-ca\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.785330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.785191 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4dj\" (UniqueName: \"kubernetes.io/projected/c254b441-da43-44c6-9bb7-6fa6775b4514-kube-api-access-nv4dj\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.785330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.785302 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.785330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.785321 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-metrics-certs\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.786655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.786631 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c254b441-da43-44c6-9bb7-6fa6775b4514-openshift-service-ca\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.786858 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:12:16.786838 4911 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-serving-cert: secret "maas-api-e2e-inf-a-784444-serving-cert" not found Nov 01 16:12:16.786965 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:12:16.786914 4911 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls podName:c254b441-da43-44c6-9bb7-6fa6775b4514 nodeName:}" failed. No retries permitted until 2026-11-01 16:12:17.28689471 +0000 UTC m=+1477.431349333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls") pod "maas-api-e2e-inf-a-784444-57f58cd57f-649bw" (UID: "c254b441-da43-44c6-9bb7-6fa6775b4514") : secret "maas-api-e2e-inf-a-784444-serving-cert" not found Nov 01 16:12:16.798528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.798496 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4dj\" (UniqueName: \"kubernetes.io/projected/c254b441-da43-44c6-9bb7-6fa6775b4514-kube-api-access-nv4dj\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:16.805846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:16.805813 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-metrics-certs\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:17.298805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:17.298767 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:17.302209 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:17.302174 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls\") pod \"maas-api-e2e-inf-a-784444-57f58cd57f-649bw\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:17.564821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:17.564331 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:17.897974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:17.897939 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw"] Nov 01 16:12:17.898477 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:12:17.898453 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc254b441_da43_44c6_9bb7_6fa6775b4514.slice/crio-ea00cc1fdc1c8d523d1e64920272e3c1b8e1c3b071a5731d6dae338d978af034 WatchSource:0}: Error finding container ea00cc1fdc1c8d523d1e64920272e3c1b8e1c3b071a5731d6dae338d978af034: Status 404 returned error can't find the container with id ea00cc1fdc1c8d523d1e64920272e3c1b8e1c3b071a5731d6dae338d978af034 Nov 01 16:12:18.894087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:18.881676 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" event={"ID":"c254b441-da43-44c6-9bb7-6fa6775b4514","Type":"ContainerStarted","Data":"6eddc5d6e48ea29994890039e499804ca6521ee1c29eff01b12030c146a19b72"} Nov 01 16:12:18.894087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:18.881760 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" event={"ID":"c254b441-da43-44c6-9bb7-6fa6775b4514","Type":"ContainerStarted","Data":"ea00cc1fdc1c8d523d1e64920272e3c1b8e1c3b071a5731d6dae338d978af034"} Nov 01 16:12:18.894087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:18.883141 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:18.933061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:18.919723 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" podStartSLOduration=2.452271293 podStartE2EDuration="2.91970061s" totalImagesPullingTime="467.429317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:18.904053498 +0000 UTC m=+1479.048508132" watchObservedRunningTime="2026-11-01 16:12:18.91970061 +0000 UTC m=+1479.064155253" Nov 01 16:12:25.910306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:25.910218 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:12:40.436784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:40.436317 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:12:40.457545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:12:40.457514 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:12:40.870158 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:12:40.870120 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:13:06.014729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:06.014690 4911 generic.go:356] "Generic (PLEG): container finished" podID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerID="2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e" exitCode=0 Nov 01 16:13:06.015238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:06.014763 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" event={"ID":"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2","Type":"ContainerDied","Data":"2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e"} Nov 01 16:13:06.015339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:06.015323 4911 scope.go:122] "RemoveContainer" containerID="2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e" Nov 01 16:13:06.017436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:06.017408 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.017508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:06.017483 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.017556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:06.017518 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:07.026967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:07.026923 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" event={"ID":"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2","Type":"ContainerStarted","Data":"81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b"} Nov 01 16:13:07.316931 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:07.316844 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:13:07.318438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:07.318407 4911 patch_prober.go:29] interesting pod/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 01 16:13:07.318555 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:07.318469 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 01 16:13:08.317645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:08.317611 4911 patch_prober.go:29] interesting pod/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 01 16:13:08.319206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:08.319141 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 01 16:13:09.317592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:09.317557 4911 patch_prober.go:29] interesting pod/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 01 16:13:09.317770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:09.317624 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 01 16:13:09.341705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:09.341671 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:13:10.317019 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:10.316982 4911 patch_prober.go:29] interesting pod/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 01 16:13:10.317245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:10.317051 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 01 16:13:10.903183 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:13:10.903146 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:13:11.318469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:11.318431 4911 patch_prober.go:29] interesting pod/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 01 16:13:11.318689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:11.318509 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 01 16:13:12.317791 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:12.317755 4911 patch_prober.go:29] interesting pod/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 01 16:13:12.318211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:12.317819 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 01 16:13:12.868128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:12.868092 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l"] Nov 01 16:13:12.873535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:12.873503 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:12.879980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:12.879952 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-99afca-serving-cert\"" Nov 01 16:13:12.882201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:12.882174 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l"] Nov 01 16:13:13.015880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.015827 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-metrics-certs\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.016129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.015908 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phz9b\" (UniqueName: \"kubernetes.io/projected/73011146-c739-4bff-bbea-9a44dd98b493-kube-api-access-phz9b\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.016129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.016024 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73011146-c739-4bff-bbea-9a44dd98b493-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.016351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.016317 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-maas-api-tls\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.117565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.117529 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73011146-c739-4bff-bbea-9a44dd98b493-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.117861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.117840 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-maas-api-tls\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.117956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.117940 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-metrics-certs\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.118027 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.118011 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phz9b\" (UniqueName: \"kubernetes.io/projected/73011146-c739-4bff-bbea-9a44dd98b493-kube-api-access-phz9b\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.118402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.118356 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73011146-c739-4bff-bbea-9a44dd98b493-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.123559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.123537 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-metrics-certs\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.123559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.123551 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-maas-api-tls\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.128894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.128865 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phz9b\" (UniqueName: \"kubernetes.io/projected/73011146-c739-4bff-bbea-9a44dd98b493-kube-api-access-phz9b\") pod \"maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.193266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.193231 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:13.327977 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.327922 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:13:13.336289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.330697 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:13:13.631814 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:13:13.631463 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73011146_c739_4bff_bbea_9a44dd98b493.slice/crio-699757bf0c43d0ca3e79bfc5397b8c6308a50da81b097823c5bd1bb1cd4169d3 WatchSource:0}: Error finding container 699757bf0c43d0ca3e79bfc5397b8c6308a50da81b097823c5bd1bb1cd4169d3: Status 404 returned error can't find the container with id 699757bf0c43d0ca3e79bfc5397b8c6308a50da81b097823c5bd1bb1cd4169d3 Nov 01 16:13:13.685299 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:13.681311 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l"] Nov 01 16:13:14.177060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:14.175615 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" event={"ID":"73011146-c739-4bff-bbea-9a44dd98b493","Type":"ContainerStarted","Data":"699757bf0c43d0ca3e79bfc5397b8c6308a50da81b097823c5bd1bb1cd4169d3"} Nov 01 16:13:15.187324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:15.187282 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" event={"ID":"73011146-c739-4bff-bbea-9a44dd98b493","Type":"ContainerStarted","Data":"fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6"} Nov 01 16:13:15.187750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:15.187378 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:15.205895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:15.205836 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" podStartSLOduration=2.626712841 podStartE2EDuration="3.205814481s" totalImagesPullingTime="579.10164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:15.203726506 +0000 UTC m=+1535.348181150" watchObservedRunningTime="2026-11-01 16:13:15.205814481 +0000 UTC m=+1535.350269125" Nov 01 16:13:21.208343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:21.208309 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:13:40.869985 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:13:40.869940 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:13:55.451321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.451224 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8"] Nov 01 16:13:55.457567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.457537 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.466622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.465980 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-99afca\"/\"openshift-service-ca.crt\"" Nov 01 16:13:55.466928 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.466908 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-99afca\"/\"kube-root-ca.crt\"" Nov 01 16:13:55.467425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.467402 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-99afca\"/\"default-dockercfg-rd6bb\"" Nov 01 16:13:55.467602 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.467479 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-99afca\"/\"test-model-99afca-kserve-self-signed-certs\"" Nov 01 16:13:55.468386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.468363 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8"] Nov 01 16:13:55.603064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.602301 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vzr\" (UniqueName: \"kubernetes.io/projected/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-kube-api-access-64vzr\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.603064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.602431 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-dshm\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.603064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.602682 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tls-certs\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.603064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.602809 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-model-cache\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.603064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.602950 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-home\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.603064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.602996 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tmp-dir\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.707987 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-model-cache\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.708148 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-home\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.708192 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tmp-dir\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.708340 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vzr\" (UniqueName: \"kubernetes.io/projected/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-kube-api-access-64vzr\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.708386 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-dshm\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.709476 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-home\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.709616 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tls-certs\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.710550 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-model-cache\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.718116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.711194 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tmp-dir\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.745740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.745705 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tls-certs\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.747353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.747320 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vzr\" (UniqueName: \"kubernetes.io/projected/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-kube-api-access-64vzr\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.749967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.749941 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-dshm\") pod \"test-model-99afca-kserve-58f55d5b69-qkkn8\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:55.794536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:55.794500 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:13:56.141303 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:13:56.141277 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4aa6807_c041_4c0e_b2ab_8bcbbc69033f.slice/crio-9c4f163f99607ddf79e9d9d2a47dbc4ee91b225b38ae4f6252965eddcb475bc3 WatchSource:0}: Error finding container 9c4f163f99607ddf79e9d9d2a47dbc4ee91b225b38ae4f6252965eddcb475bc3: Status 404 returned error can't find the container with id 9c4f163f99607ddf79e9d9d2a47dbc4ee91b225b38ae4f6252965eddcb475bc3 Nov 01 16:13:56.144259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:56.144226 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8"] Nov 01 16:13:56.863820 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:56.863762 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" event={"ID":"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f","Type":"ContainerStarted","Data":"d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29"} Nov 01 16:13:56.863820 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:56.863822 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" event={"ID":"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f","Type":"ContainerStarted","Data":"9c4f163f99607ddf79e9d9d2a47dbc4ee91b225b38ae4f6252965eddcb475bc3"} Nov 01 16:13:56.883556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:56.883497 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" podStartSLOduration=1.883482447 podStartE2EDuration="1.883482447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:56.882122942 +0000 UTC m=+1577.026577585" watchObservedRunningTime="2026-11-01 16:13:56.883482447 +0000 UTC m=+1577.027937088" Nov 01 16:13:57.795910 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:13:57.795873 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:14:00.671262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.671153 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d"] Nov 01 16:14:00.676953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.676925 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.692544 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.692405 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d"] Nov 01 16:14:00.693697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.693672 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2001b0\"/\"default-dockercfg-mx8p7\"" Nov 01 16:14:00.693860 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.693773 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2001b0\"/\"test-model-2001b0-kserve-self-signed-certs\"" Nov 01 16:14:00.716698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.716660 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-home\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.716901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.716770 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-tmp-dir\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.716901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.716826 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-model-cache\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.716901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.716880 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-dshm\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.717444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.717408 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b69ff4-60ba-477a-b077-406f0f88b4d6-tls-certs\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.717577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.717461 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxm9\" (UniqueName: \"kubernetes.io/projected/77b69ff4-60ba-477a-b077-406f0f88b4d6-kube-api-access-ftxm9\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.720486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.720462 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2001b0\"/\"kube-root-ca.crt\"" Nov 01 16:14:00.720650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.720495 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2001b0\"/\"openshift-service-ca.crt\"" Nov 01 16:14:00.819171 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.819137 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-home\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.819366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.819227 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-tmp-dir\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.819366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.819274 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-model-cache\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.819366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.819324 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-dshm\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.819635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.819607 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b69ff4-60ba-477a-b077-406f0f88b4d6-tls-certs\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.819715 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.819633 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-home\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.819715 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.819659 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxm9\" (UniqueName: \"kubernetes.io/projected/77b69ff4-60ba-477a-b077-406f0f88b4d6-kube-api-access-ftxm9\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.820569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.820521 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-model-cache\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.820930 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.820908 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-tmp-dir\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.823641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.823611 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b69ff4-60ba-477a-b077-406f0f88b4d6-tls-certs\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.823770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.823748 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-dshm\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:00.828464 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:00.828434 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxm9\" (UniqueName: \"kubernetes.io/projected/77b69ff4-60ba-477a-b077-406f0f88b4d6-kube-api-access-ftxm9\") pod \"test-model-2001b0-kserve-7645fb85fc-mg82d\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:01.001636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:01.001603 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:01.481469 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:14:01.481430 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77b69ff4_60ba_477a_b077_406f0f88b4d6.slice/crio-d7734c4a21c962cb7ac38fa5f5770fb6fd36bc1fa9c1dc34f610e5a52543bb2e WatchSource:0}: Error finding container d7734c4a21c962cb7ac38fa5f5770fb6fd36bc1fa9c1dc34f610e5a52543bb2e: Status 404 returned error can't find the container with id d7734c4a21c962cb7ac38fa5f5770fb6fd36bc1fa9c1dc34f610e5a52543bb2e Nov 01 16:14:01.484868 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:01.484837 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d"] Nov 01 16:14:01.935028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:01.934936 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" event={"ID":"77b69ff4-60ba-477a-b077-406f0f88b4d6","Type":"ContainerStarted","Data":"0333d40d0f58b999b30702b866679ce82ac7bd3a1c0b759edee132285abb8b14"} Nov 01 16:14:01.935028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:01.934984 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" event={"ID":"77b69ff4-60ba-477a-b077-406f0f88b4d6","Type":"ContainerStarted","Data":"d7734c4a21c962cb7ac38fa5f5770fb6fd36bc1fa9c1dc34f610e5a52543bb2e"} Nov 01 16:14:01.957871 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:01.957818 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" podStartSLOduration=1.957804858 podStartE2EDuration="1.957804858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:14:01.953141072 +0000 UTC m=+1582.097595713" watchObservedRunningTime="2026-11-01 16:14:01.957804858 +0000 UTC m=+1582.102259535" Nov 01 16:14:02.003403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:02.003364 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:05.795731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:05.795692 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:14:05.809269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:05.809237 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:14:05.998832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:05.998796 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:14:10.869569 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:14:10.869527 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:14:11.019059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:11.008603 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:11.144923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:11.144606 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:11.169822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:11.169786 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:14:18.793089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.785091 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v"] Nov 01 16:14:18.860267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.860234 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v"] Nov 01 16:14:18.860586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.860569 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:18.874610 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.873700 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-10b00a\"/\"openshift-service-ca.crt\"" Nov 01 16:14:18.889105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.886010 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-10b00a\"/\"test-model-10b00a-kserve-self-signed-certs\"" Nov 01 16:14:18.889326 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.887223 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-10b00a\"/\"kube-root-ca.crt\"" Nov 01 16:14:18.889426 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.887245 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-10b00a\"/\"default-dockercfg-blrq6\"" Nov 01 16:14:18.908106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.907453 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fr9\" (UniqueName: \"kubernetes.io/projected/2c2e75c6-d218-4d2c-a851-32ee17c455df-kube-api-access-h8fr9\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:18.908106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.907539 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-home\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:18.908106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.907590 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-tmp-dir\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:18.908106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.907736 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-model-cache\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:18.908106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.907766 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2e75c6-d218-4d2c-a851-32ee17c455df-tls-certs\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:18.908106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:18.908061 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-dshm\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010014 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-model-cache\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010313 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010121 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2e75c6-d218-4d2c-a851-32ee17c455df-tls-certs\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010426 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010403 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-dshm\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010459 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-model-cache\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010520 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fr9\" (UniqueName: \"kubernetes.io/projected/2c2e75c6-d218-4d2c-a851-32ee17c455df-kube-api-access-h8fr9\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010618 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010580 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-home\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010632 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-tmp-dir\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.010983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.010962 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-home\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.011167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.011143 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-tmp-dir\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.014973 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.014947 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2e75c6-d218-4d2c-a851-32ee17c455df-tls-certs\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.015884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.015866 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-dshm\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.021101 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.021073 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fr9\" (UniqueName: \"kubernetes.io/projected/2c2e75c6-d218-4d2c-a851-32ee17c455df-kube-api-access-h8fr9\") pod \"test-model-10b00a-kserve-57f6c46867-25n7v\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.191120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.187301 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:19.619805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:19.619769 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v"] Nov 01 16:14:19.619921 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:14:19.619843 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c2e75c6_d218_4d2c_a851_32ee17c455df.slice/crio-cf10e13c8e60417337651fcd956b61c5e036972add3c16b5914d491a7040e01c WatchSource:0}: Error finding container cf10e13c8e60417337651fcd956b61c5e036972add3c16b5914d491a7040e01c: Status 404 returned error can't find the container with id cf10e13c8e60417337651fcd956b61c5e036972add3c16b5914d491a7040e01c Nov 01 16:14:20.274694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:20.274650 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" event={"ID":"2c2e75c6-d218-4d2c-a851-32ee17c455df","Type":"ContainerStarted","Data":"7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55"} Nov 01 16:14:20.275664 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:20.274706 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" event={"ID":"2c2e75c6-d218-4d2c-a851-32ee17c455df","Type":"ContainerStarted","Data":"cf10e13c8e60417337651fcd956b61c5e036972add3c16b5914d491a7040e01c"} Nov 01 16:14:20.294607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:20.294545 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" podStartSLOduration=2.294526786 podStartE2EDuration="2.294526786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:14:20.290631984 +0000 UTC m=+1600.435086625" watchObservedRunningTime="2026-11-01 16:14:20.294526786 +0000 UTC m=+1600.438981429" Nov 01 16:14:21.189924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:21.189891 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:29.190308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:29.190271 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:29.205358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:29.205328 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:29.377305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:14:29.377270 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:14:40.888888 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:14:40.888850 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:15:00.155683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.155644 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr"] Nov 01 16:15:00.168352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.168217 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc"] Nov 01 16:15:00.168639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.168318 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.178139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.178108 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.214242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.209017 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:15:00.219986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.219950 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr"] Nov 01 16:15:00.239467 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.239374 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc"] Nov 01 16:15:00.255774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.255750 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf"] Nov 01 16:15:00.256145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.256121 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" containerID="cri-o://6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce" gracePeriod=30 Nov 01 16:15:00.280243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.280199 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.280700 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.280676 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d811349-4144-4d51-a15e-5f81feb6b7ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.280885 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.280867 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttms\" (UniqueName: \"kubernetes.io/projected/1d811349-4144-4d51-a15e-5f81feb6b7ed-kube-api-access-zttms\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.282610 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.282579 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9ls\" (UniqueName: \"kubernetes.io/projected/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-kube-api-access-mb9ls\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.384112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.384063 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9ls\" (UniqueName: \"kubernetes.io/projected/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-kube-api-access-mb9ls\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.384315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.384275 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.384450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.384433 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d811349-4144-4d51-a15e-5f81feb6b7ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.384525 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.384469 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttms\" (UniqueName: \"kubernetes.io/projected/1d811349-4144-4d51-a15e-5f81feb6b7ed-kube-api-access-zttms\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.384937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.384915 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.385120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.385095 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d811349-4144-4d51-a15e-5f81feb6b7ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.396497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.396464 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9ls\" (UniqueName: \"kubernetes.io/projected/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-kube-api-access-mb9ls\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.396656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.396593 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttms\" (UniqueName: \"kubernetes.io/projected/1d811349-4144-4d51-a15e-5f81feb6b7ed-kube-api-access-zttms\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.432866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.432773 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt"] Nov 01 16:15:00.433307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.433268 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" containerID="cri-o://bc227bd9ac7841ffb8169336a9457a922f3c253bc346afd01e06f8b4d4f8556a" gracePeriod=30 Nov 01 16:15:00.544293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.544257 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:00.566440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.566401 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:00.625548 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.625185 4911 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:15:00.625548 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.625261 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:15:00.695600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.695459 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7"] Nov 01 16:15:00.697108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.696480 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" containerID="cri-o://95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4" gracePeriod=30 Nov 01 16:15:00.871423 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:00.871387 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d811349_4144_4d51_a15e_5f81feb6b7ed.slice/crio-8a3bcd5490051dc347f25db982e06cf36ce239eebdd09b0bb8c12596608914a7 WatchSource:0}: Error finding container 8a3bcd5490051dc347f25db982e06cf36ce239eebdd09b0bb8c12596608914a7: Status 404 returned error can't find the container with id 8a3bcd5490051dc347f25db982e06cf36ce239eebdd09b0bb8c12596608914a7 Nov 01 16:15:00.873467 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.873432 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr"] Nov 01 16:15:00.888150 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:00.888118 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc"] Nov 01 16:15:00.890900 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:00.890875 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd22252_dc4e_41bd_8dfe_2e0dc3fb4ec2.slice/crio-a64cccb7d667a93bebee0b954f85af716ee1cd32cd0cb59e13729b3ec210593c WatchSource:0}: Error finding container a64cccb7d667a93bebee0b954f85af716ee1cd32cd0cb59e13729b3ec210593c: Status 404 returned error can't find the container with id a64cccb7d667a93bebee0b954f85af716ee1cd32cd0cb59e13729b3ec210593c Nov 01 16:15:01.735310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:01.735175 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerStarted","Data":"b235f92e2d16cabe41ec478e915ea7338fcfef2353dd6cc23d7371ecc962912b"} Nov 01 16:15:01.735310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:01.735242 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerStarted","Data":"8a3bcd5490051dc347f25db982e06cf36ce239eebdd09b0bb8c12596608914a7"} Nov 01 16:15:01.737190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:01.737157 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerStarted","Data":"4835d0f56b327b3369274c30f2370ce3b90b02fbb27ea2bdd6aabb10711caa4a"} Nov 01 16:15:01.737330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:01.737214 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerStarted","Data":"a64cccb7d667a93bebee0b954f85af716ee1cd32cd0cb59e13729b3ec210593c"} Nov 01 16:15:01.761830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:01.761771 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" podStartSLOduration=1.761752924 podStartE2EDuration="1.761752924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:01.756065375 +0000 UTC m=+1641.900520016" watchObservedRunningTime="2026-11-01 16:15:01.761752924 +0000 UTC m=+1641.906207566" Nov 01 16:15:01.778217 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:01.778152 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" podStartSLOduration=1.778133998 podStartE2EDuration="1.778133998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:01.773985116 +0000 UTC m=+1641.918439771" watchObservedRunningTime="2026-11-01 16:15:01.778133998 +0000 UTC m=+1641.922588640" Nov 01 16:15:02.747355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:02.747317 4911 generic.go:356] "Generic (PLEG): container finished" podID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerID="4835d0f56b327b3369274c30f2370ce3b90b02fbb27ea2bdd6aabb10711caa4a" exitCode=60 Nov 01 16:15:02.747854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:02.747353 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerDied","Data":"4835d0f56b327b3369274c30f2370ce3b90b02fbb27ea2bdd6aabb10711caa4a"} Nov 01 16:15:02.747854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:02.747801 4911 scope.go:122] "RemoveContainer" containerID="4835d0f56b327b3369274c30f2370ce3b90b02fbb27ea2bdd6aabb10711caa4a" Nov 01 16:15:02.749613 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:02.749579 4911 generic.go:356] "Generic (PLEG): container finished" podID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerID="b235f92e2d16cabe41ec478e915ea7338fcfef2353dd6cc23d7371ecc962912b" exitCode=60 Nov 01 16:15:02.749744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:02.749621 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerDied","Data":"b235f92e2d16cabe41ec478e915ea7338fcfef2353dd6cc23d7371ecc962912b"} Nov 01 16:15:02.749989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:02.749972 4911 scope.go:122] "RemoveContainer" containerID="b235f92e2d16cabe41ec478e915ea7338fcfef2353dd6cc23d7371ecc962912b" Nov 01 16:15:03.760938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:03.760900 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerStarted","Data":"af9b1093770902ec4656b68caa1bb3b49c290411ade0d1ade0c1f5d018206e19"} Nov 01 16:15:03.763362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:03.763332 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerStarted","Data":"eb521b003f6603c368e2025bfa888d8d915ca0e6a4c2589ca5c874e232c686c9"} Nov 01 16:15:04.777640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.777605 4911 generic.go:356] "Generic (PLEG): container finished" podID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerID="af9b1093770902ec4656b68caa1bb3b49c290411ade0d1ade0c1f5d018206e19" exitCode=60 Nov 01 16:15:04.778133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.777696 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerDied","Data":"af9b1093770902ec4656b68caa1bb3b49c290411ade0d1ade0c1f5d018206e19"} Nov 01 16:15:04.778133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.777772 4911 scope.go:122] "RemoveContainer" containerID="b235f92e2d16cabe41ec478e915ea7338fcfef2353dd6cc23d7371ecc962912b" Nov 01 16:15:04.778133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.777980 4911 scope.go:122] "RemoveContainer" containerID="af9b1093770902ec4656b68caa1bb3b49c290411ade0d1ade0c1f5d018206e19" Nov 01 16:15:04.778483 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:04.778448 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr_odh-ai-gateway-infra(1d811349-4144-4d51-a15e-5f81feb6b7ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" Nov 01 16:15:04.780158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.780131 4911 generic.go:356] "Generic (PLEG): container finished" podID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerID="eb521b003f6603c368e2025bfa888d8d915ca0e6a4c2589ca5c874e232c686c9" exitCode=60 Nov 01 16:15:04.780270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.780203 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerDied","Data":"eb521b003f6603c368e2025bfa888d8d915ca0e6a4c2589ca5c874e232c686c9"} Nov 01 16:15:04.780500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.780484 4911 scope.go:122] "RemoveContainer" containerID="eb521b003f6603c368e2025bfa888d8d915ca0e6a4c2589ca5c874e232c686c9" Nov 01 16:15:04.780838 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:04.780817 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc_odh-ai-gateway-infra(9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" Nov 01 16:15:04.792297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:04.792275 4911 scope.go:122] "RemoveContainer" containerID="4835d0f56b327b3369274c30f2370ce3b90b02fbb27ea2bdd6aabb10711caa4a" Nov 01 16:15:05.591030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.590997 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 16:15:05.636764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.636731 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg"] Nov 01 16:15:05.638412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.637979 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" Nov 01 16:15:05.638412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.638031 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" Nov 01 16:15:05.638412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.638086 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" Nov 01 16:15:05.638412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.638096 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" Nov 01 16:15:05.638884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.638483 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" Nov 01 16:15:05.638884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.638503 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" containerName="istio-proxy" Nov 01 16:15:05.651098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.649979 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg"] Nov 01 16:15:05.651098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.650149 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.665571 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.665529 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-99afca-openshift-default-dockercfg-7gj25\"" Nov 01 16:15:05.688445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.688415 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c728783-d532-41ca-9ea3-b50340b395cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c728783-d532-41ca-9ea3-b50340b395cd-istiod-ca-cert\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.688600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.688502 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-istio-token\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-istio-token\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.688686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.688669 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-data\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.688748 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.688711 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-certs\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.688802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.688752 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c728783-d532-41ca-9ea3-b50340b395cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c728783-d532-41ca-9ea3-b50340b395cd-istio-podinfo\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.688861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.688827 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c728783-d532-41ca-9ea3-b50340b395cd-istiod-ca-cert" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:05.688861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.688838 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-socket\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.689287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.689168 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-socket" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.689287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.689212 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-kube-api-access-jgblt\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-kube-api-access-jgblt\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.689287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.689258 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-data" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.689287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.689281 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-credential-socket\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.689603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.689450 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-envoy\") pod \"4c728783-d532-41ca-9ea3-b50340b395cd\" (UID: \"4c728783-d532-41ca-9ea3-b50340b395cd\") " Nov 01 16:15:05.689603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.689541 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-certs" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.689603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.689554 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-credential-socket" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.690360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.690335 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c728783-d532-41ca-9ea3-b50340b395cd-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.690465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.690363 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.690465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.690377 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.690465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.690391 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.690465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.690404 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.693698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.693672 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-kube-api-access-jgblt" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "kube-api-access-jgblt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:05.694193 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.694169 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-envoy" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.694303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.694230 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-istio-token" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:05.695816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.695793 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c728783-d532-41ca-9ea3-b50340b395cd-istio-podinfo" pod "4c728783-d532-41ca-9ea3-b50340b395cd" (UID: "4c728783-d532-41ca-9ea3-b50340b395cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:15:05.791708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.791670 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.791766 4911 scope.go:122] "RemoveContainer" containerID="eb521b003f6603c368e2025bfa888d8d915ca0e6a4c2589ca5c874e232c686c9" Nov 01 16:15:05.792179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.791853 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.791899 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csd7q\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-kube-api-access-csd7q\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792179 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:05.792164 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc_odh-ai-gateway-infra(9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" Nov 01 16:15:05.792437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792261 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792391 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792498 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792604 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792579 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792621 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792734 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.792900 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792880 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c728783-d532-41ca-9ea3-b50340b395cd-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.792900 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792900 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgblt\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-kube-api-access-jgblt\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.793118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792914 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c728783-d532-41ca-9ea3-b50340b395cd-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.793118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.792926 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c728783-d532-41ca-9ea3-b50340b395cd-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.794366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.794345 4911 generic.go:356] "Generic (PLEG): container finished" podID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerID="bc227bd9ac7841ffb8169336a9457a922f3c253bc346afd01e06f8b4d4f8556a" exitCode=0 Nov 01 16:15:05.794472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.794392 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" event={"ID":"16a71a41-49b4-4557-8cea-e9f9f01f7709","Type":"ContainerDied","Data":"bc227bd9ac7841ffb8169336a9457a922f3c253bc346afd01e06f8b4d4f8556a"} Nov 01 16:15:05.796663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.796496 4911 generic.go:356] "Generic (PLEG): container finished" podID="4c728783-d532-41ca-9ea3-b50340b395cd" containerID="6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce" exitCode=0 Nov 01 16:15:05.796663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.796559 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" Nov 01 16:15:05.796663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.796589 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" event={"ID":"4c728783-d532-41ca-9ea3-b50340b395cd","Type":"ContainerDied","Data":"6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce"} Nov 01 16:15:05.796663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.796639 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf" event={"ID":"4c728783-d532-41ca-9ea3-b50340b395cd","Type":"ContainerDied","Data":"8dbb525730365a0b17b6e1744162d2fb692ee5c55873f2c20d9501a837d53766"} Nov 01 16:15:05.796937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.796669 4911 scope.go:122] "RemoveContainer" containerID="6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce" Nov 01 16:15:05.799594 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.799277 4911 scope.go:122] "RemoveContainer" containerID="af9b1093770902ec4656b68caa1bb3b49c290411ade0d1ade0c1f5d018206e19" Nov 01 16:15:05.799594 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:05.799510 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr_odh-ai-gateway-infra(1d811349-4144-4d51-a15e-5f81feb6b7ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" Nov 01 16:15:05.825575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.825544 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:15:05.827976 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.827764 4911 scope.go:122] "RemoveContainer" containerID="e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d" Nov 01 16:15:05.842555 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.842527 4911 scope.go:122] "RemoveContainer" containerID="6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce" Nov 01 16:15:05.847628 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:05.842997 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce\": container with ID starting with 6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce not found: ID does not exist" containerID="6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce" Nov 01 16:15:05.847628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.844172 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce"} err="failed to get container status \"6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce\": rpc error: code = NotFound desc = could not find container \"6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce\": container with ID starting with 6a4d0eb943d5784f0ede048a398da980f77f0cff4ab07c28edc6b18a22b492ce not found: ID does not exist" Nov 01 16:15:05.847628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.844219 4911 scope.go:122] "RemoveContainer" containerID="e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d" Nov 01 16:15:05.847628 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:05.846841 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d\": container with ID starting with e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d not found: ID does not exist" containerID="e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d" Nov 01 16:15:05.847628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.846881 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d"} err="failed to get container status \"e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d\": rpc error: code = NotFound desc = could not find container \"e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d\": container with ID starting with e02d1f0573d203090584213cbfa82e7b4d91e452212ccdec25a3cae017ec1e7d not found: ID does not exist" Nov 01 16:15:05.864783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.864712 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf"] Nov 01 16:15:05.879184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.879126 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r96sf"] Nov 01 16:15:05.911468 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.910293 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6"] Nov 01 16:15:05.919072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.917553 4911 cpu_manager.go:414] "containerMap: removing container" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" Nov 01 16:15:05.919072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.917584 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" Nov 01 16:15:05.919072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.917821 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" containerName="istio-proxy" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921304 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921376 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921425 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921520 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921624 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921805 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921876 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csd7q\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-kube-api-access-csd7q\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.921978 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.922103 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.923066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.922562 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.926414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.926322 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.927492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.926772 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.932468 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.932395 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.939054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.936155 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.956327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.942816 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6"] Nov 01 16:15:05.956327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.942942 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:05.970921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.961464 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csd7q\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-kube-api-access-csd7q\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.970921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.966833 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9e044c-serving-cert\"" Nov 01 16:15:05.970921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.967314 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.970921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.967673 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.976230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.972516 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.980521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.980338 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:05.987375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:05.983020 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c728783-d532-41ca-9ea3-b50340b395cd" path="/var/lib/kubelet/pods/4c728783-d532-41ca-9ea3-b50340b395cd/volumes" Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024226 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-socket\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024309 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-credential-socket\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024453 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a71a41-49b4-4557-8cea-e9f9f01f7709-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a71a41-49b4-4557-8cea-e9f9f01f7709-istiod-ca-cert\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024601 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-kube-api-access-8ggg5\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-kube-api-access-8ggg5\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024675 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-envoy\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024768 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-podinfo\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024798 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-token\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-token\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024876 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-data\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.026016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.024950 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-certs\") pod \"16a71a41-49b4-4557-8cea-e9f9f01f7709\" (UID: \"16a71a41-49b4-4557-8cea-e9f9f01f7709\") " Nov 01 16:15:06.027072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.026909 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-certs" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.027957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.027929 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-socket" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.028142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.028124 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-credential-socket" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.028691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.028649 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a71a41-49b4-4557-8cea-e9f9f01f7709-istiod-ca-cert" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:06.029819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.029573 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-data" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.032719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.032694 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-podinfo" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:15:06.034306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.034229 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-token" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.034728 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.034704 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-envoy" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.036540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.036514 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-kube-api-access-8ggg5" pod "16a71a41-49b4-4557-8cea-e9f9f01f7709" (UID: "16a71a41-49b4-4557-8cea-e9f9f01f7709"). InnerVolumeSpecName "kube-api-access-8ggg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.101486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.101281 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138144 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-socket\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138212 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istiod-ca-cert\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138274 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-envoy\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138351 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-certs\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138427 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-data\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138474 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-token\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-token\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138609 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-kube-api-access-hcxht\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-kube-api-access-hcxht\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138691 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-podinfo\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138780 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-credential-socket\") pod \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\" (UID: \"0dbd1cae-eb9b-4e44-aeea-731867bb217f\") " Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138955 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b92acb9-5d34-4f35-9926-cdb0334f5c80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.138990 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-metrics-certs\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139169 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88j2n\" (UniqueName: \"kubernetes.io/projected/7b92acb9-5d34-4f35-9926-cdb0334f5c80-kube-api-access-88j2n\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139291 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139444 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139457 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139466 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a71a41-49b4-4557-8cea-e9f9f01f7709-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139475 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ggg5\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-kube-api-access-8ggg5\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139484 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139497 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139508 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139516 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139526 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a71a41-49b4-4557-8cea-e9f9f01f7709-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.139943 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-socket" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.140347 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istiod-ca-cert" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.140585 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-certs" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.141146 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-data" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.143927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.141420 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-credential-socket" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.145518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.144209 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-podinfo" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:15:06.148870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.148709 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-token" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.154513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.154461 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-envoy" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.157492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.157457 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-kube-api-access-hcxht" pod "0dbd1cae-eb9b-4e44-aeea-731867bb217f" (UID: "0dbd1cae-eb9b-4e44-aeea-731867bb217f"). InnerVolumeSpecName "kube-api-access-hcxht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.168958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.168927 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z"] Nov 01 16:15:06.170989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.170962 4911 cpu_manager.go:414] "containerMap: removing container" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" Nov 01 16:15:06.171174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.171156 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" Nov 01 16:15:06.171508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.171490 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerName="istio-proxy" Nov 01 16:15:06.192172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.191974 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll"] Nov 01 16:15:06.192703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.192149 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.199245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.199122 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp"] Nov 01 16:15:06.199526 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.199215 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.214179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.212807 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4"] Nov 01 16:15:06.214783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.214644 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.223142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.222613 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm"] Nov 01 16:15:06.223142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.222733 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.231134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.231105 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht"] Nov 01 16:15:06.233070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.233013 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.248698 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfhc\" (UniqueName: \"kubernetes.io/projected/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-kube-api-access-xmfhc\") pod \"maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.248829 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.248880 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88j2n\" (UniqueName: \"kubernetes.io/projected/7b92acb9-5d34-4f35-9926-cdb0334f5c80-kube-api-access-88j2n\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249077 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249344 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b92acb9-5d34-4f35-9926-cdb0334f5c80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249413 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-metrics-certs\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249657 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcxht\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-kube-api-access-hcxht\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249679 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249694 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249722 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249737 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249751 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249764 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249793 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.249807 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbd1cae-eb9b-4e44-aeea-731867bb217f-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.250620 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp"] Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.252085 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.257195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.252700 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b92acb9-5d34-4f35-9926-cdb0334f5c80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.261299 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.261257 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb"] Nov 01 16:15:06.262169 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.261348 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.267724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.267697 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.270739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.270717 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs"] Nov 01 16:15:06.271125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.270933 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.272777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.272749 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88j2n\" (UniqueName: \"kubernetes.io/projected/7b92acb9-5d34-4f35-9926-cdb0334f5c80-kube-api-access-88j2n\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.272878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.272755 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-metrics-certs\") pod \"maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.276987 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.276965 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6"] Nov 01 16:15:06.277137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.277113 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.282880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.282858 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z"] Nov 01 16:15:06.282984 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.282902 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll"] Nov 01 16:15:06.282984 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.282928 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4"] Nov 01 16:15:06.283078 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.283017 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.291349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.291326 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht"] Nov 01 16:15:06.298020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.297965 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp"] Nov 01 16:15:06.307392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.307357 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm"] Nov 01 16:15:06.316250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.316209 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6"] Nov 01 16:15:06.321357 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.321333 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp"] Nov 01 16:15:06.321483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.321401 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:06.337208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.337150 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs"] Nov 01 16:15:06.339295 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:06.339243 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d53d81_84e1_47d0_b62e_29fe3e8ef937.slice/crio-46fb2fec24f3e3b65ff60c51df1c09ce3722c13714726423c9fb00a2c883b6ae WatchSource:0}: Error finding container 46fb2fec24f3e3b65ff60c51df1c09ce3722c13714726423c9fb00a2c883b6ae: Status 404 returned error can't find the container with id 46fb2fec24f3e3b65ff60c51df1c09ce3722c13714726423c9fb00a2c883b6ae Nov 01 16:15:06.344501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.344480 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb"] Nov 01 16:15:06.345795 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.345762 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.345872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.345828 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.345872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.345867 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.351318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351286 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnb8r\" (UniqueName: \"kubernetes.io/projected/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-kube-api-access-tnb8r\") pod \"maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.351498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351473 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqsls\" (UniqueName: \"kubernetes.io/projected/d5321715-5daf-4973-9218-0ce2ac3a0282-kube-api-access-tqsls\") pod \"maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.351649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351509 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eef4f46-6006-4038-8209-77ed34168412-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.351649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351582 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5321715-5daf-4973-9218-0ce2ac3a0282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.351880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351652 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vvf\" (UniqueName: \"kubernetes.io/projected/a7be1a72-b700-49aa-8b9c-d80cdc39168a-kube-api-access-b4vvf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.351954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351918 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.351954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351945 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrld\" (UniqueName: \"kubernetes.io/projected/7f085329-c40e-48bb-9978-fe2f8fb00695-kube-api-access-kfrld\") pod \"maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.352074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.351994 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.353145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.352265 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed858f9a-835c-4c94-9866-5f28cdf2550b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.353145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.352945 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcktf\" (UniqueName: \"kubernetes.io/projected/ed858f9a-835c-4c94-9866-5f28cdf2550b-kube-api-access-bcktf\") pod \"maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.353145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.352997 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7be1a72-b700-49aa-8b9c-d80cdc39168a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.353145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.353083 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422af011-c613-4443-8047-d86cb44cd89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.353403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.353166 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfhc\" (UniqueName: \"kubernetes.io/projected/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-kube-api-access-xmfhc\") pod \"maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.353403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.353383 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.353561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.353490 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxj7\" (UniqueName: \"kubernetes.io/projected/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-kube-api-access-dlxj7\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.354175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.353768 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085329-c40e-48bb-9978-fe2f8fb00695-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.354175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.353912 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twn92\" (UniqueName: \"kubernetes.io/projected/422af011-c613-4443-8047-d86cb44cd89f-kube-api-access-twn92\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.354175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.353983 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mnp\" (UniqueName: \"kubernetes.io/projected/9eef4f46-6006-4038-8209-77ed34168412-kube-api-access-t4mnp\") pod \"maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.354175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.354072 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.363427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.363403 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfhc\" (UniqueName: \"kubernetes.io/projected/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-kube-api-access-xmfhc\") pod \"maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.432559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.432371 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg"] Nov 01 16:15:06.455749 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.455715 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7be1a72-b700-49aa-8b9c-d80cdc39168a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.455933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.455783 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422af011-c613-4443-8047-d86cb44cd89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.455933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.455915 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxj7\" (UniqueName: \"kubernetes.io/projected/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-kube-api-access-dlxj7\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.456090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.455967 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085329-c40e-48bb-9978-fe2f8fb00695-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.456090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456011 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twn92\" (UniqueName: \"kubernetes.io/projected/422af011-c613-4443-8047-d86cb44cd89f-kube-api-access-twn92\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.456090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456075 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mnp\" (UniqueName: \"kubernetes.io/projected/9eef4f46-6006-4038-8209-77ed34168412-kube-api-access-t4mnp\") pod \"maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456129 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnb8r\" (UniqueName: \"kubernetes.io/projected/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-kube-api-access-tnb8r\") pod \"maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456161 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqsls\" (UniqueName: \"kubernetes.io/projected/d5321715-5daf-4973-9218-0ce2ac3a0282-kube-api-access-tqsls\") pod \"maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456189 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eef4f46-6006-4038-8209-77ed34168412-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456235 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5321715-5daf-4973-9218-0ce2ac3a0282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456335 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vvf\" (UniqueName: \"kubernetes.io/projected/a7be1a72-b700-49aa-8b9c-d80cdc39168a-kube-api-access-b4vvf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456419 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456445 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrld\" (UniqueName: \"kubernetes.io/projected/7f085329-c40e-48bb-9978-fe2f8fb00695-kube-api-access-kfrld\") pod \"maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456496 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456554 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef06614d-8817-4af9-a659-2287b72ed0d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.456702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456671 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed858f9a-835c-4c94-9866-5f28cdf2550b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.457257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456722 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjwn\" (UniqueName: \"kubernetes.io/projected/ef06614d-8817-4af9-a659-2287b72ed0d0-kube-api-access-nmjwn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.457257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.456759 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcktf\" (UniqueName: \"kubernetes.io/projected/ed858f9a-835c-4c94-9866-5f28cdf2550b-kube-api-access-bcktf\") pod \"maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.461004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.457796 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422af011-c613-4443-8047-d86cb44cd89f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.461004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.459838 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed858f9a-835c-4c94-9866-5f28cdf2550b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.461004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.460190 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5321715-5daf-4973-9218-0ce2ac3a0282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.461004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.460956 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eef4f46-6006-4038-8209-77ed34168412-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.463575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.462113 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085329-c40e-48bb-9978-fe2f8fb00695-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.463575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.462766 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.463575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.463519 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7be1a72-b700-49aa-8b9c-d80cdc39168a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.464679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.464659 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.481364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.481240 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrld\" (UniqueName: \"kubernetes.io/projected/7f085329-c40e-48bb-9978-fe2f8fb00695-kube-api-access-kfrld\") pod \"maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.482299 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.482003 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcktf\" (UniqueName: \"kubernetes.io/projected/ed858f9a-835c-4c94-9866-5f28cdf2550b-kube-api-access-bcktf\") pod \"maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.483689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.483617 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqsls\" (UniqueName: \"kubernetes.io/projected/d5321715-5daf-4973-9218-0ce2ac3a0282-kube-api-access-tqsls\") pod \"maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.483800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.483691 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vvf\" (UniqueName: \"kubernetes.io/projected/a7be1a72-b700-49aa-8b9c-d80cdc39168a-kube-api-access-b4vvf\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.484407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.484318 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twn92\" (UniqueName: \"kubernetes.io/projected/422af011-c613-4443-8047-d86cb44cd89f-kube-api-access-twn92\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.484703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.484676 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mnp\" (UniqueName: \"kubernetes.io/projected/9eef4f46-6006-4038-8209-77ed34168412-kube-api-access-t4mnp\") pod \"maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.501074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.501026 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxj7\" (UniqueName: \"kubernetes.io/projected/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-kube-api-access-dlxj7\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.501669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.501555 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnb8r\" (UniqueName: \"kubernetes.io/projected/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-kube-api-access-tnb8r\") pod \"maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.528777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.528182 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:06.533733 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.533552 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:06.543082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.543007 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:06.574723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.560566 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjwn\" (UniqueName: \"kubernetes.io/projected/ef06614d-8817-4af9-a659-2287b72ed0d0-kube-api-access-nmjwn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.574723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.561554 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef06614d-8817-4af9-a659-2287b72ed0d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.574723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.567753 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef06614d-8817-4af9-a659-2287b72ed0d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.591211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.589817 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjwn\" (UniqueName: \"kubernetes.io/projected/ef06614d-8817-4af9-a659-2287b72ed0d0-kube-api-access-nmjwn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.595707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.594524 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:06.595707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.595092 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:06.595707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.595305 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:06.620828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.620800 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:06.631383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.631002 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:06.692786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.684088 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6"] Nov 01 16:15:06.695110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.694928 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:15:06.772967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.769423 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:06.820421 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.820371 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:06.852158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.852066 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" event={"ID":"19d53d81-84e1-47d0-b62e-29fe3e8ef937","Type":"ContainerStarted","Data":"e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994"} Nov 01 16:15:06.852473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.852448 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" event={"ID":"19d53d81-84e1-47d0-b62e-29fe3e8ef937","Type":"ContainerStarted","Data":"46fb2fec24f3e3b65ff60c51df1c09ce3722c13714726423c9fb00a2c883b6ae"} Nov 01 16:15:06.857199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.857094 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" event={"ID":"7b92acb9-5d34-4f35-9926-cdb0334f5c80","Type":"ContainerStarted","Data":"80e615439181cb4910f8224ce16356fd430ef4ac4f0814b0b8049ecf0d5db4bd"} Nov 01 16:15:06.864059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.861420 4911 generic.go:356] "Generic (PLEG): container finished" podID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" containerID="95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4" exitCode=0 Nov 01 16:15:06.864059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.861583 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" event={"ID":"0dbd1cae-eb9b-4e44-aeea-731867bb217f","Type":"ContainerDied","Data":"95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4"} Nov 01 16:15:06.864059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.861631 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" event={"ID":"0dbd1cae-eb9b-4e44-aeea-731867bb217f","Type":"ContainerDied","Data":"e15ed581cfa417f3c6d961c1a30b3b9eb2b67f415f3e6ba50111286e1afbb7e5"} Nov 01 16:15:06.864059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.861661 4911 scope.go:122] "RemoveContainer" containerID="95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4" Nov 01 16:15:06.864059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.861851 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7" Nov 01 16:15:06.891994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.888862 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podStartSLOduration=65.888841536 podStartE2EDuration="1m5.888841536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:06.882708067 +0000 UTC m=+1647.027162709" watchObservedRunningTime="2026-11-01 16:15:06.888841536 +0000 UTC m=+1647.033296179" Nov 01 16:15:06.936649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.926658 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" event={"ID":"16a71a41-49b4-4557-8cea-e9f9f01f7709","Type":"ContainerDied","Data":"51bcac5c2e5937a8626c2e0c6c63f4fc6c2cd5fcefcc9ab98e4e5601a685b984"} Nov 01 16:15:06.936649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.926829 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt" Nov 01 16:15:06.954613 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.954585 4911 scope.go:122] "RemoveContainer" containerID="95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4" Nov 01 16:15:06.967126 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:06.965557 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4\": container with ID starting with 95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4 not found: ID does not exist" containerID="95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4" Nov 01 16:15:06.967126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.965613 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4"} err="failed to get container status \"95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4\": rpc error: code = NotFound desc = could not find container \"95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4\": container with ID starting with 95747332e5a3f201b1a4d7a96ef127a98892832dd93f9bf0533aa8b1a19568a4 not found: ID does not exist" Nov 01 16:15:06.967126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.965646 4911 scope.go:122] "RemoveContainer" containerID="bc227bd9ac7841ffb8169336a9457a922f3c253bc346afd01e06f8b4d4f8556a" Nov 01 16:15:06.989663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:06.982909 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:07.000174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.000094 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7"] Nov 01 16:15:07.043162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.043098 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-tjhs7"] Nov 01 16:15:07.065687 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.065655 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt"] Nov 01 16:15:07.073980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.073943 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-qdrlt"] Nov 01 16:15:07.105358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.102310 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z"] Nov 01 16:15:07.112001 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:07.111946 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422af011_c613_4443_8047_d86cb44cd89f.slice/crio-0844dbab4ed29902390ddf0c9304718c1a9f4e9a734cdc8765dcdcb0742dba8e WatchSource:0}: Error finding container 0844dbab4ed29902390ddf0c9304718c1a9f4e9a734cdc8765dcdcb0742dba8e: Status 404 returned error can't find the container with id 0844dbab4ed29902390ddf0c9304718c1a9f4e9a734cdc8765dcdcb0742dba8e Nov 01 16:15:07.398246 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:07.398212 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5321715_5daf_4973_9218_0ce2ac3a0282.slice/crio-c44f487bc46c486e0c18e7b4bb5f8e0464660a48a3c80ccc6be36ea430fc14df WatchSource:0}: Error finding container c44f487bc46c486e0c18e7b4bb5f8e0464660a48a3c80ccc6be36ea430fc14df: Status 404 returned error can't find the container with id c44f487bc46c486e0c18e7b4bb5f8e0464660a48a3c80ccc6be36ea430fc14df Nov 01 16:15:07.424663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.424636 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp"] Nov 01 16:15:07.429017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.428983 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht"] Nov 01 16:15:07.516848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.516818 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll"] Nov 01 16:15:07.526612 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.523437 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp"] Nov 01 16:15:07.532694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.532312 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb"] Nov 01 16:15:07.842144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.842097 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm"] Nov 01 16:15:07.935675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:07.928733 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6"] Nov 01 16:15:08.004761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.004725 4911 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 01 16:15:08.005024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.004995 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 01 16:15:08.146276 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.145729 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbd1cae-eb9b-4e44-aeea-731867bb217f" path="/var/lib/kubelet/pods/0dbd1cae-eb9b-4e44-aeea-731867bb217f/volumes" Nov 01 16:15:08.146567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.146540 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a71a41-49b4-4557-8cea-e9f9f01f7709" path="/var/lib/kubelet/pods/16a71a41-49b4-4557-8cea-e9f9f01f7709/volumes" Nov 01 16:15:08.147279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.147259 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs"] Nov 01 16:15:08.147473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.147458 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4"] Nov 01 16:15:08.147603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.147585 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerStarted","Data":"8e50bd155c626db33558c5015410b5053f70073350f9b2a1b235fc5d6f0eed53"} Nov 01 16:15:08.150435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.148978 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" event={"ID":"9eef4f46-6006-4038-8209-77ed34168412","Type":"ContainerStarted","Data":"28e444890d59ac8046f52e1308e52d541d84d936d2df184911e9667b0713fab0"} Nov 01 16:15:08.173157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.173101 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" event={"ID":"d5321715-5daf-4973-9218-0ce2ac3a0282","Type":"ContainerStarted","Data":"c44f487bc46c486e0c18e7b4bb5f8e0464660a48a3c80ccc6be36ea430fc14df"} Nov 01 16:15:08.175615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.175570 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerStarted","Data":"32433749eaf1f995d70088df1a10abf787f10073fd042212ea14e13c6ea25376"} Nov 01 16:15:08.183204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.183169 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerStarted","Data":"7af076d0d14fba5f44c0f8cdc890640198208faef43c62fc556115b0d4318cd0"} Nov 01 16:15:08.184760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.184731 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerStarted","Data":"0844dbab4ed29902390ddf0c9304718c1a9f4e9a734cdc8765dcdcb0742dba8e"} Nov 01 16:15:08.186398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.186365 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerStarted","Data":"38bfd394499e86e05a046b449d60fc2c762479ede393d80807912ee8c2817522"} Nov 01 16:15:08.188518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.188493 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" event={"ID":"7b92acb9-5d34-4f35-9926-cdb0334f5c80","Type":"ContainerStarted","Data":"d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3"} Nov 01 16:15:08.189022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.188975 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:08.199056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.197524 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerStarted","Data":"ae05f064a9d6444fc634a1c2cc6f8b32db0256b1219f78574f9474fc98a6961e"} Nov 01 16:15:08.201395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.201370 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerStarted","Data":"4777665d3476271bab27262f98a298176c6dc0f17306e0ee52d96de0900e880e"} Nov 01 16:15:08.221203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.221155 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" podStartSLOduration=8.221141913 podStartE2EDuration="8.221141913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:08.220360586 +0000 UTC m=+1648.364815228" watchObservedRunningTime="2026-11-01 16:15:08.221141913 +0000 UTC m=+1648.365596555" Nov 01 16:15:08.243160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.243108 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" podStartSLOduration=55.637183935 podStartE2EDuration="56.243094053s" totalImagesPullingTime="605.910118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:08.241729967 +0000 UTC m=+1648.386184607" watchObservedRunningTime="2026-11-01 16:15:08.243094053 +0000 UTC m=+1648.387548695" Nov 01 16:15:08.989904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.983774 4911 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 01 16:15:08.995187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:08.995130 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 01 16:15:09.215076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.214994 4911 generic.go:356] "Generic (PLEG): container finished" podID="9eef4f46-6006-4038-8209-77ed34168412" containerID="b244f49cfbce28b911fcc0469dd96e6e5c1841dc190b039b20a8b36d3fef5f19" exitCode=60 Nov 01 16:15:09.215196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.215067 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" event={"ID":"9eef4f46-6006-4038-8209-77ed34168412","Type":"ContainerDied","Data":"b244f49cfbce28b911fcc0469dd96e6e5c1841dc190b039b20a8b36d3fef5f19"} Nov 01 16:15:09.216116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.215515 4911 scope.go:122] "RemoveContainer" containerID="b244f49cfbce28b911fcc0469dd96e6e5c1841dc190b039b20a8b36d3fef5f19" Nov 01 16:15:09.217290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.217267 4911 generic.go:356] "Generic (PLEG): container finished" podID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerID="676e10ebabcde003934c45e72f1fafbff3a1e3af8bf848b1b22b435362c4d27c" exitCode=60 Nov 01 16:15:09.217435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.217317 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" event={"ID":"d5321715-5daf-4973-9218-0ce2ac3a0282","Type":"ContainerDied","Data":"676e10ebabcde003934c45e72f1fafbff3a1e3af8bf848b1b22b435362c4d27c"} Nov 01 16:15:09.217666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.217649 4911 scope.go:122] "RemoveContainer" containerID="676e10ebabcde003934c45e72f1fafbff3a1e3af8bf848b1b22b435362c4d27c" Nov 01 16:15:09.219182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.219157 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerStarted","Data":"0a8e8019d3591af9c0d4b138d4be395db5586c0f1b7d653b309ee497afe2f657"} Nov 01 16:15:09.221164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.221135 4911 generic.go:356] "Generic (PLEG): container finished" podID="422af011-c613-4443-8047-d86cb44cd89f" containerID="7af076d0d14fba5f44c0f8cdc890640198208faef43c62fc556115b0d4318cd0" exitCode=60 Nov 01 16:15:09.221262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.221240 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerDied","Data":"7af076d0d14fba5f44c0f8cdc890640198208faef43c62fc556115b0d4318cd0"} Nov 01 16:15:09.221576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.221473 4911 scope.go:122] "RemoveContainer" containerID="7af076d0d14fba5f44c0f8cdc890640198208faef43c62fc556115b0d4318cd0" Nov 01 16:15:09.223004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.222984 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerStarted","Data":"05b5adf3e585fe8b1ba9253d985ee4ea332bf7d5fbda3be7fa51a1ff9a216594"} Nov 01 16:15:09.224691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.224670 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" event={"ID":"7f085329-c40e-48bb-9978-fe2f8fb00695","Type":"ContainerStarted","Data":"e5d030ad9ac0b3ce1e158ecf789ae3c3142df64930c37d6acafecb4bdbe238ae"} Nov 01 16:15:09.226329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.226305 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerStarted","Data":"54af6b00eb0734229517de9f3b54fcf631b49f75fc1ec881f86815e518f806da"} Nov 01 16:15:09.226418 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.226345 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerStarted","Data":"3c2bc1f80570ff45244fab8d438b9d714ff088d7fbb63d19c817485b0eb92913"} Nov 01 16:15:09.228090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.227994 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerStarted","Data":"4124a45805ed187b5f51a6e9a08b7c20dace6a01c29c3980e7941c47eedee6b9"} Nov 01 16:15:09.230550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.230007 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerStarted","Data":"feb6cc523b335806fe4c3ce87eb6d4b22c24fc705a8a5a68fb8a6f077b5e092b"} Nov 01 16:15:09.233236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.233212 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerStarted","Data":"1da0ce5caee5286169dd26c1c80c47408629a06d29ff080ebe80e476ced3fe79"} Nov 01 16:15:09.301173 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.300537 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" podStartSLOduration=9.300517682 podStartE2EDuration="9.300517682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.277362548 +0000 UTC m=+1649.421817190" watchObservedRunningTime="2026-11-01 16:15:09.300517682 +0000 UTC m=+1649.444972520" Nov 01 16:15:09.302325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.302281 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" podStartSLOduration=9.302265121 podStartE2EDuration="9.302265121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.295754701 +0000 UTC m=+1649.440209349" watchObservedRunningTime="2026-11-01 16:15:09.302265121 +0000 UTC m=+1649.446719762" Nov 01 16:15:09.314990 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.314945 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" podStartSLOduration=9.31493302 podStartE2EDuration="9.31493302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.313909422 +0000 UTC m=+1649.458364065" watchObservedRunningTime="2026-11-01 16:15:09.31493302 +0000 UTC m=+1649.459387661" Nov 01 16:15:09.333798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.333751 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" podStartSLOduration=9.333737765 podStartE2EDuration="9.333737765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.330382979 +0000 UTC m=+1649.474837622" watchObservedRunningTime="2026-11-01 16:15:09.333737765 +0000 UTC m=+1649.478192443" Nov 01 16:15:09.380098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.373854 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" podStartSLOduration=9.373833251 podStartE2EDuration="9.373833251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.347384369 +0000 UTC m=+1649.491839010" watchObservedRunningTime="2026-11-01 16:15:09.373833251 +0000 UTC m=+1649.518287897" Nov 01 16:15:09.439898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.439839 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" podStartSLOduration=9.439821026 podStartE2EDuration="9.439821026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.430803036 +0000 UTC m=+1649.575257677" watchObservedRunningTime="2026-11-01 16:15:09.439821026 +0000 UTC m=+1649.584275687" Nov 01 16:15:09.494819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:09.494760 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" podStartSLOduration=9.494740783 podStartE2EDuration="9.494740783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:09.447251315 +0000 UTC m=+1649.591705956" watchObservedRunningTime="2026-11-01 16:15:09.494740783 +0000 UTC m=+1649.639195456" Nov 01 16:15:10.001706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.001670 4911 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 01 16:15:10.002097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.001729 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 01 16:15:10.247856 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.247817 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" event={"ID":"d5321715-5daf-4973-9218-0ce2ac3a0282","Type":"ContainerStarted","Data":"e5641ed028844d5a25264fbbeafea7c264b79a0178b45d58d384761045027110"} Nov 01 16:15:10.262310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.262178 4911 generic.go:356] "Generic (PLEG): container finished" podID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerID="0a8e8019d3591af9c0d4b138d4be395db5586c0f1b7d653b309ee497afe2f657" exitCode=60 Nov 01 16:15:10.272248 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.272132 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerDied","Data":"0a8e8019d3591af9c0d4b138d4be395db5586c0f1b7d653b309ee497afe2f657"} Nov 01 16:15:10.272503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.272464 4911 scope.go:122] "RemoveContainer" containerID="0a8e8019d3591af9c0d4b138d4be395db5586c0f1b7d653b309ee497afe2f657" Nov 01 16:15:10.282979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.282636 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerStarted","Data":"6f6f5683591e6045d45eb0550505c727b5f2b2f03e09cf23453ad5d8210b69fd"} Nov 01 16:15:10.285563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.285540 4911 generic.go:356] "Generic (PLEG): container finished" podID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerID="05b5adf3e585fe8b1ba9253d985ee4ea332bf7d5fbda3be7fa51a1ff9a216594" exitCode=60 Nov 01 16:15:10.285713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.285698 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerDied","Data":"05b5adf3e585fe8b1ba9253d985ee4ea332bf7d5fbda3be7fa51a1ff9a216594"} Nov 01 16:15:10.285952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.285928 4911 scope.go:122] "RemoveContainer" containerID="05b5adf3e585fe8b1ba9253d985ee4ea332bf7d5fbda3be7fa51a1ff9a216594" Nov 01 16:15:10.288091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.287754 4911 generic.go:356] "Generic (PLEG): container finished" podID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerID="b637f480d332693fca0c397957b64652e6865c0b4de414cf07869839d23c6bd8" exitCode=60 Nov 01 16:15:10.288091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.287796 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" event={"ID":"7f085329-c40e-48bb-9978-fe2f8fb00695","Type":"ContainerDied","Data":"b637f480d332693fca0c397957b64652e6865c0b4de414cf07869839d23c6bd8"} Nov 01 16:15:10.288431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.288133 4911 scope.go:122] "RemoveContainer" containerID="b637f480d332693fca0c397957b64652e6865c0b4de414cf07869839d23c6bd8" Nov 01 16:15:10.290089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.290065 4911 generic.go:356] "Generic (PLEG): container finished" podID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerID="54af6b00eb0734229517de9f3b54fcf631b49f75fc1ec881f86815e518f806da" exitCode=60 Nov 01 16:15:10.290192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.290158 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerDied","Data":"54af6b00eb0734229517de9f3b54fcf631b49f75fc1ec881f86815e518f806da"} Nov 01 16:15:10.290459 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.290441 4911 scope.go:122] "RemoveContainer" containerID="54af6b00eb0734229517de9f3b54fcf631b49f75fc1ec881f86815e518f806da" Nov 01 16:15:10.292547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.292526 4911 generic.go:356] "Generic (PLEG): container finished" podID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerID="4124a45805ed187b5f51a6e9a08b7c20dace6a01c29c3980e7941c47eedee6b9" exitCode=60 Nov 01 16:15:10.292766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.292744 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerDied","Data":"4124a45805ed187b5f51a6e9a08b7c20dace6a01c29c3980e7941c47eedee6b9"} Nov 01 16:15:10.293661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.293641 4911 scope.go:122] "RemoveContainer" containerID="4124a45805ed187b5f51a6e9a08b7c20dace6a01c29c3980e7941c47eedee6b9" Nov 01 16:15:10.295666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.295640 4911 generic.go:356] "Generic (PLEG): container finished" podID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerID="feb6cc523b335806fe4c3ce87eb6d4b22c24fc705a8a5a68fb8a6f077b5e092b" exitCode=60 Nov 01 16:15:10.295828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.295807 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerDied","Data":"feb6cc523b335806fe4c3ce87eb6d4b22c24fc705a8a5a68fb8a6f077b5e092b"} Nov 01 16:15:10.296286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.296267 4911 scope.go:122] "RemoveContainer" containerID="feb6cc523b335806fe4c3ce87eb6d4b22c24fc705a8a5a68fb8a6f077b5e092b" Nov 01 16:15:10.301500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.301332 4911 generic.go:356] "Generic (PLEG): container finished" podID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerID="1da0ce5caee5286169dd26c1c80c47408629a06d29ff080ebe80e476ced3fe79" exitCode=60 Nov 01 16:15:10.301500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.301433 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerDied","Data":"1da0ce5caee5286169dd26c1c80c47408629a06d29ff080ebe80e476ced3fe79"} Nov 01 16:15:10.304467 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.304444 4911 scope.go:122] "RemoveContainer" containerID="1da0ce5caee5286169dd26c1c80c47408629a06d29ff080ebe80e476ced3fe79" Nov 01 16:15:10.305180 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.305122 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" event={"ID":"9eef4f46-6006-4038-8209-77ed34168412","Type":"ContainerStarted","Data":"d940c711a6cc518dbe236304dbe2378fc3f88d50c4332a62b122f233baa7f49d"} Nov 01 16:15:10.307167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.307055 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" podStartSLOduration=10.307027988 podStartE2EDuration="10.307027988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:10.302672696 +0000 UTC m=+1650.447127338" watchObservedRunningTime="2026-11-01 16:15:10.307027988 +0000 UTC m=+1650.451482629" Nov 01 16:15:10.603103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.603050 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" podStartSLOduration=10.60302229 podStartE2EDuration="10.60302229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:10.600984951 +0000 UTC m=+1650.745439593" watchObservedRunningTime="2026-11-01 16:15:10.60302229 +0000 UTC m=+1650.747476931" Nov 01 16:15:10.905793 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:10.905757 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:15:10.983086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.982711 4911 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 01 16:15:10.983086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:10.982781 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 01 16:15:11.330956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.330924 4911 generic.go:356] "Generic (PLEG): container finished" podID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerID="e5641ed028844d5a25264fbbeafea7c264b79a0178b45d58d384761045027110" exitCode=60 Nov 01 16:15:11.331619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.331593 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" event={"ID":"d5321715-5daf-4973-9218-0ce2ac3a0282","Type":"ContainerDied","Data":"e5641ed028844d5a25264fbbeafea7c264b79a0178b45d58d384761045027110"} Nov 01 16:15:11.331773 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.331758 4911 scope.go:122] "RemoveContainer" containerID="676e10ebabcde003934c45e72f1fafbff3a1e3af8bf848b1b22b435362c4d27c" Nov 01 16:15:11.333510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.332835 4911 scope.go:122] "RemoveContainer" containerID="e5641ed028844d5a25264fbbeafea7c264b79a0178b45d58d384761045027110" Nov 01 16:15:11.334198 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:11.334172 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp_odh-ai-gateway-infra(d5321715-5daf-4973-9218-0ce2ac3a0282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" Nov 01 16:15:11.337950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.337891 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerStarted","Data":"ac491dde7bb3c22c12892dc8a659c955bcefe8c941c83987e8bbeeca500fc354"} Nov 01 16:15:11.342643 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.341766 4911 generic.go:356] "Generic (PLEG): container finished" podID="422af011-c613-4443-8047-d86cb44cd89f" containerID="6f6f5683591e6045d45eb0550505c727b5f2b2f03e09cf23453ad5d8210b69fd" exitCode=60 Nov 01 16:15:11.342643 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.341877 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerDied","Data":"6f6f5683591e6045d45eb0550505c727b5f2b2f03e09cf23453ad5d8210b69fd"} Nov 01 16:15:11.342643 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.342291 4911 scope.go:122] "RemoveContainer" containerID="6f6f5683591e6045d45eb0550505c727b5f2b2f03e09cf23453ad5d8210b69fd" Nov 01 16:15:11.342643 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:11.342593 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z_odh-ai-gateway-infra(422af011-c613-4443-8047-d86cb44cd89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" podUID="422af011-c613-4443-8047-d86cb44cd89f" Nov 01 16:15:11.346775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.346756 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerStarted","Data":"a49e819a1307a3476a426a302cbd4b54c6dc341ba0a265bab2e5e981d4d52596"} Nov 01 16:15:11.397062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.378624 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" event={"ID":"7f085329-c40e-48bb-9978-fe2f8fb00695","Type":"ContainerStarted","Data":"f84c12d7e673113bec31a1380024b1b7cefb894d2d2c17f37bf4bb44e3bf8525"} Nov 01 16:15:11.407452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.407396 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerStarted","Data":"2317819da2a7530b7d551802d60e8f18b04dbbcdd64649b85869e860db4f0610"} Nov 01 16:15:11.410823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.410795 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerStarted","Data":"377d9c8c9ee70fd8bce0b395f82fc3bce70cfd4858ce54bbed5f08beac38ebe5"} Nov 01 16:15:11.411722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.411593 4911 scope.go:122] "RemoveContainer" containerID="7af076d0d14fba5f44c0f8cdc890640198208faef43c62fc556115b0d4318cd0" Nov 01 16:15:11.417438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.417417 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerStarted","Data":"ebe46f8b9cae74ba05eafd54fbd88484543f7eb4f9481c402ac1a9261ead3a60"} Nov 01 16:15:11.425013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.424906 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerStarted","Data":"73cbce92fa987abcd70c82ccf17bbae6e50fbb371879ba9000fe0b6972d752c2"} Nov 01 16:15:11.428605 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.428581 4911 generic.go:356] "Generic (PLEG): container finished" podID="9eef4f46-6006-4038-8209-77ed34168412" containerID="d940c711a6cc518dbe236304dbe2378fc3f88d50c4332a62b122f233baa7f49d" exitCode=60 Nov 01 16:15:11.428882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.428861 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" event={"ID":"9eef4f46-6006-4038-8209-77ed34168412","Type":"ContainerDied","Data":"d940c711a6cc518dbe236304dbe2378fc3f88d50c4332a62b122f233baa7f49d"} Nov 01 16:15:11.429415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.429396 4911 scope.go:122] "RemoveContainer" containerID="d940c711a6cc518dbe236304dbe2378fc3f88d50c4332a62b122f233baa7f49d" Nov 01 16:15:11.432537 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:11.432488 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht_odh-ai-gateway-infra(9eef4f46-6006-4038-8209-77ed34168412)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" podUID="9eef4f46-6006-4038-8209-77ed34168412" Nov 01 16:15:11.433619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.433559 4911 scope.go:122] "RemoveContainer" containerID="b244f49cfbce28b911fcc0469dd96e6e5c1841dc190b039b20a8b36d3fef5f19" Nov 01 16:15:11.988999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.986890 4911 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 01 16:15:11.988999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:11.986949 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 01 16:15:12.441664 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.441570 4911 generic.go:356] "Generic (PLEG): container finished" podID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerID="a49e819a1307a3476a426a302cbd4b54c6dc341ba0a265bab2e5e981d4d52596" exitCode=60 Nov 01 16:15:12.442117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.441655 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerDied","Data":"a49e819a1307a3476a426a302cbd4b54c6dc341ba0a265bab2e5e981d4d52596"} Nov 01 16:15:12.442117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.441730 4911 scope.go:122] "RemoveContainer" containerID="05b5adf3e585fe8b1ba9253d985ee4ea332bf7d5fbda3be7fa51a1ff9a216594" Nov 01 16:15:12.442117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.441984 4911 scope.go:122] "RemoveContainer" containerID="a49e819a1307a3476a426a302cbd4b54c6dc341ba0a265bab2e5e981d4d52596" Nov 01 16:15:12.442482 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.442452 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb_odh-ai-gateway-infra(ef06614d-8817-4af9-a659-2287b72ed0d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" Nov 01 16:15:12.444108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.444081 4911 generic.go:356] "Generic (PLEG): container finished" podID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerID="f84c12d7e673113bec31a1380024b1b7cefb894d2d2c17f37bf4bb44e3bf8525" exitCode=60 Nov 01 16:15:12.444247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.444155 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" event={"ID":"7f085329-c40e-48bb-9978-fe2f8fb00695","Type":"ContainerDied","Data":"f84c12d7e673113bec31a1380024b1b7cefb894d2d2c17f37bf4bb44e3bf8525"} Nov 01 16:15:12.444413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.444394 4911 scope.go:122] "RemoveContainer" containerID="f84c12d7e673113bec31a1380024b1b7cefb894d2d2c17f37bf4bb44e3bf8525" Nov 01 16:15:12.444795 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.444768 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs_odh-ai-gateway-infra(7f085329-c40e-48bb-9978-fe2f8fb00695)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" Nov 01 16:15:12.446508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.446489 4911 generic.go:356] "Generic (PLEG): container finished" podID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerID="2317819da2a7530b7d551802d60e8f18b04dbbcdd64649b85869e860db4f0610" exitCode=60 Nov 01 16:15:12.446599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.446544 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerDied","Data":"2317819da2a7530b7d551802d60e8f18b04dbbcdd64649b85869e860db4f0610"} Nov 01 16:15:12.446839 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.446824 4911 scope.go:122] "RemoveContainer" containerID="2317819da2a7530b7d551802d60e8f18b04dbbcdd64649b85869e860db4f0610" Nov 01 16:15:12.447303 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.447267 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4_odh-ai-gateway-infra(a7be1a72-b700-49aa-8b9c-d80cdc39168a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" Nov 01 16:15:12.448990 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.448971 4911 generic.go:356] "Generic (PLEG): container finished" podID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerID="377d9c8c9ee70fd8bce0b395f82fc3bce70cfd4858ce54bbed5f08beac38ebe5" exitCode=60 Nov 01 16:15:12.449126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.449092 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerDied","Data":"377d9c8c9ee70fd8bce0b395f82fc3bce70cfd4858ce54bbed5f08beac38ebe5"} Nov 01 16:15:12.458121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.449276 4911 scope.go:122] "RemoveContainer" containerID="377d9c8c9ee70fd8bce0b395f82fc3bce70cfd4858ce54bbed5f08beac38ebe5" Nov 01 16:15:12.458121 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.449540 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm_odh-ai-gateway-infra(dfda58ef-e7b3-47cb-b357-ba60d84a04cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" Nov 01 16:15:12.479810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.479763 4911 generic.go:356] "Generic (PLEG): container finished" podID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerID="ebe46f8b9cae74ba05eafd54fbd88484543f7eb4f9481c402ac1a9261ead3a60" exitCode=60 Nov 01 16:15:12.479968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.479861 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerDied","Data":"ebe46f8b9cae74ba05eafd54fbd88484543f7eb4f9481c402ac1a9261ead3a60"} Nov 01 16:15:12.481080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.480470 4911 scope.go:122] "RemoveContainer" containerID="ebe46f8b9cae74ba05eafd54fbd88484543f7eb4f9481c402ac1a9261ead3a60" Nov 01 16:15:12.481080 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.481003 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll_odh-ai-gateway-infra(ed858f9a-835c-4c94-9866-5f28cdf2550b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" Nov 01 16:15:12.485883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.485851 4911 generic.go:356] "Generic (PLEG): container finished" podID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerID="73cbce92fa987abcd70c82ccf17bbae6e50fbb371879ba9000fe0b6972d752c2" exitCode=60 Nov 01 16:15:12.486015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.485985 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerDied","Data":"73cbce92fa987abcd70c82ccf17bbae6e50fbb371879ba9000fe0b6972d752c2"} Nov 01 16:15:12.489274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.489250 4911 scope.go:122] "RemoveContainer" containerID="73cbce92fa987abcd70c82ccf17bbae6e50fbb371879ba9000fe0b6972d752c2" Nov 01 16:15:12.489744 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.489722 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6_odh-ai-gateway-infra(0324204c-7d4f-4644-aa70-39e7e8a0e7e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" Nov 01 16:15:12.500933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.500798 4911 scope.go:122] "RemoveContainer" containerID="b637f480d332693fca0c397957b64652e6865c0b4de414cf07869839d23c6bd8" Nov 01 16:15:12.506138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.506115 4911 scope.go:122] "RemoveContainer" containerID="d940c711a6cc518dbe236304dbe2378fc3f88d50c4332a62b122f233baa7f49d" Nov 01 16:15:12.506603 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.506580 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht_odh-ai-gateway-infra(9eef4f46-6006-4038-8209-77ed34168412)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" podUID="9eef4f46-6006-4038-8209-77ed34168412" Nov 01 16:15:12.515024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.514994 4911 scope.go:122] "RemoveContainer" containerID="e5641ed028844d5a25264fbbeafea7c264b79a0178b45d58d384761045027110" Nov 01 16:15:12.515807 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.515778 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp_odh-ai-gateway-infra(d5321715-5daf-4973-9218-0ce2ac3a0282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" Nov 01 16:15:12.521639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.519081 4911 generic.go:356] "Generic (PLEG): container finished" podID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerID="ac491dde7bb3c22c12892dc8a659c955bcefe8c941c83987e8bbeeca500fc354" exitCode=60 Nov 01 16:15:12.521639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.519166 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerDied","Data":"ac491dde7bb3c22c12892dc8a659c955bcefe8c941c83987e8bbeeca500fc354"} Nov 01 16:15:12.521639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.519742 4911 scope.go:122] "RemoveContainer" containerID="ac491dde7bb3c22c12892dc8a659c955bcefe8c941c83987e8bbeeca500fc354" Nov 01 16:15:12.521639 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.520493 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp_odh-ai-gateway-infra(d2ba2926-33bf-4a15-b31d-f40ddd6c72c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" Nov 01 16:15:12.524857 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.524023 4911 scope.go:122] "RemoveContainer" containerID="6f6f5683591e6045d45eb0550505c727b5f2b2f03e09cf23453ad5d8210b69fd" Nov 01 16:15:12.524857 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:12.524493 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z_odh-ai-gateway-infra(422af011-c613-4443-8047-d86cb44cd89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" podUID="422af011-c613-4443-8047-d86cb44cd89f" Nov 01 16:15:12.530362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.530340 4911 scope.go:122] "RemoveContainer" containerID="54af6b00eb0734229517de9f3b54fcf631b49f75fc1ec881f86815e518f806da" Nov 01 16:15:12.591817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.591675 4911 scope.go:122] "RemoveContainer" containerID="4124a45805ed187b5f51a6e9a08b7c20dace6a01c29c3980e7941c47eedee6b9" Nov 01 16:15:12.620159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.620134 4911 scope.go:122] "RemoveContainer" containerID="feb6cc523b335806fe4c3ce87eb6d4b22c24fc705a8a5a68fb8a6f077b5e092b" Nov 01 16:15:12.645676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.645643 4911 scope.go:122] "RemoveContainer" containerID="1da0ce5caee5286169dd26c1c80c47408629a06d29ff080ebe80e476ced3fe79" Nov 01 16:15:12.720462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.720417 4911 scope.go:122] "RemoveContainer" containerID="0a8e8019d3591af9c0d4b138d4be395db5586c0f1b7d653b309ee497afe2f657" Nov 01 16:15:12.982822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.982739 4911 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 01 16:15:12.982822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:12.982810 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 01 16:15:13.539778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.539740 4911 scope.go:122] "RemoveContainer" containerID="ac491dde7bb3c22c12892dc8a659c955bcefe8c941c83987e8bbeeca500fc354" Nov 01 16:15:13.540524 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:13.540385 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp_odh-ai-gateway-infra(d2ba2926-33bf-4a15-b31d-f40ddd6c72c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" Nov 01 16:15:13.542013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.541986 4911 scope.go:122] "RemoveContainer" containerID="a49e819a1307a3476a426a302cbd4b54c6dc341ba0a265bab2e5e981d4d52596" Nov 01 16:15:13.542536 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:13.542493 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb_odh-ai-gateway-infra(ef06614d-8817-4af9-a659-2287b72ed0d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" Nov 01 16:15:13.544766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.544739 4911 scope.go:122] "RemoveContainer" containerID="f84c12d7e673113bec31a1380024b1b7cefb894d2d2c17f37bf4bb44e3bf8525" Nov 01 16:15:13.545188 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:13.545162 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs_odh-ai-gateway-infra(7f085329-c40e-48bb-9978-fe2f8fb00695)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" Nov 01 16:15:13.546862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.546832 4911 scope.go:122] "RemoveContainer" containerID="2317819da2a7530b7d551802d60e8f18b04dbbcdd64649b85869e860db4f0610" Nov 01 16:15:13.548022 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:13.547995 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4_odh-ai-gateway-infra(a7be1a72-b700-49aa-8b9c-d80cdc39168a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" Nov 01 16:15:13.548899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.548877 4911 scope.go:122] "RemoveContainer" containerID="377d9c8c9ee70fd8bce0b395f82fc3bce70cfd4858ce54bbed5f08beac38ebe5" Nov 01 16:15:13.549216 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:13.549188 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm_odh-ai-gateway-infra(dfda58ef-e7b3-47cb-b357-ba60d84a04cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" Nov 01 16:15:13.551112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.551090 4911 scope.go:122] "RemoveContainer" containerID="ebe46f8b9cae74ba05eafd54fbd88484543f7eb4f9481c402ac1a9261ead3a60" Nov 01 16:15:13.551396 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:13.551375 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll_odh-ai-gateway-infra(ed858f9a-835c-4c94-9866-5f28cdf2550b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" Nov 01 16:15:13.553632 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.553597 4911 scope.go:122] "RemoveContainer" containerID="73cbce92fa987abcd70c82ccf17bbae6e50fbb371879ba9000fe0b6972d752c2" Nov 01 16:15:13.554080 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:13.554055 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6_odh-ai-gateway-infra(0324204c-7d4f-4644-aa70-39e7e8a0e7e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" Nov 01 16:15:13.972211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.972126 4911 scope.go:122] "RemoveContainer" containerID="af9b1093770902ec4656b68caa1bb3b49c290411ade0d1ade0c1f5d018206e19" Nov 01 16:15:13.989448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.989416 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:13.989936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.989911 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:13.991227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:13.991205 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:15:14.241506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:14.241473 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:15:14.568198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:14.568070 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerStarted","Data":"2a076241f69d9de1d76fec0c14c2d7fbbf4fa0b3adc741887072ae8e8f3bd81d"} Nov 01 16:15:15.580393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:15.580359 4911 generic.go:356] "Generic (PLEG): container finished" podID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerID="2a076241f69d9de1d76fec0c14c2d7fbbf4fa0b3adc741887072ae8e8f3bd81d" exitCode=60 Nov 01 16:15:15.580833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:15.580402 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerDied","Data":"2a076241f69d9de1d76fec0c14c2d7fbbf4fa0b3adc741887072ae8e8f3bd81d"} Nov 01 16:15:15.580833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:15.580498 4911 scope.go:122] "RemoveContainer" containerID="af9b1093770902ec4656b68caa1bb3b49c290411ade0d1ade0c1f5d018206e19" Nov 01 16:15:15.581224 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:15.581204 4911 scope.go:122] "RemoveContainer" containerID="2a076241f69d9de1d76fec0c14c2d7fbbf4fa0b3adc741887072ae8e8f3bd81d" Nov 01 16:15:15.581517 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:15.581491 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr_odh-ai-gateway-infra(1d811349-4144-4d51-a15e-5f81feb6b7ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" Nov 01 16:15:15.601267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:15.601238 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr"] Nov 01 16:15:15.976274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:15.976178 4911 scope.go:122] "RemoveContainer" containerID="eb521b003f6603c368e2025bfa888d8d915ca0e6a4c2589ca5c874e232c686c9" Nov 01 16:15:16.594085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.594021 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerStarted","Data":"2b3205fa5a0bc9e6b831e4ab00486a3c71f8fcf64f16ecee5a2e2937d8c397d7"} Nov 01 16:15:16.849818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.849743 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.881444 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6"] Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882443 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882461 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882485 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882490 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882501 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882507 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882676 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.882698 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:16.893315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.889971 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:16.895419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.894783 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6"] Nov 01 16:15:16.983023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.978012 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d811349-4144-4d51-a15e-5f81feb6b7ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d811349-4144-4d51-a15e-5f81feb6b7ed-maas-api-service-ca\") pod \"1d811349-4144-4d51-a15e-5f81feb6b7ed\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " Nov 01 16:15:16.983023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.978216 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d811349-4144-4d51-a15e-5f81feb6b7ed-kube-api-access-zttms\" (UniqueName: \"kubernetes.io/projected/1d811349-4144-4d51-a15e-5f81feb6b7ed-kube-api-access-zttms\") pod \"1d811349-4144-4d51-a15e-5f81feb6b7ed\" (UID: \"1d811349-4144-4d51-a15e-5f81feb6b7ed\") " Nov 01 16:15:16.983023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.978962 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa496de7-fb8d-4329-9302-b6983e105838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:16.983023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:16.979078 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv52g\" (UniqueName: \"kubernetes.io/projected/fa496de7-fb8d-4329-9302-b6983e105838-kube-api-access-sv52g\") pod \"maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:17.003085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.002537 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d811349-4144-4d51-a15e-5f81feb6b7ed-kube-api-access-zttms" pod "1d811349-4144-4d51-a15e-5f81feb6b7ed" (UID: "1d811349-4144-4d51-a15e-5f81feb6b7ed"). InnerVolumeSpecName "kube-api-access-zttms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:17.003326 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.003300 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d811349-4144-4d51-a15e-5f81feb6b7ed-maas-api-service-ca" pod "1d811349-4144-4d51-a15e-5f81feb6b7ed" (UID: "1d811349-4144-4d51-a15e-5f81feb6b7ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:17.080835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.080797 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa496de7-fb8d-4329-9302-b6983e105838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:17.080983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.080913 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv52g\" (UniqueName: \"kubernetes.io/projected/fa496de7-fb8d-4329-9302-b6983e105838-kube-api-access-sv52g\") pod \"maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:17.081070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.081031 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttms\" (UniqueName: \"kubernetes.io/projected/1d811349-4144-4d51-a15e-5f81feb6b7ed-kube-api-access-zttms\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:17.081140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.081069 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d811349-4144-4d51-a15e-5f81feb6b7ed-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:17.081779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.081661 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa496de7-fb8d-4329-9302-b6983e105838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:17.094163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.094131 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv52g\" (UniqueName: \"kubernetes.io/projected/fa496de7-fb8d-4329-9302-b6983e105838-kube-api-access-sv52g\") pod \"maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:17.215071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.214953 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:17.508052 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:17.507997 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa496de7_fb8d_4329_9302_b6983e105838.slice/crio-7ca4366b4f0438928f20f36039d2d58c8406608d147c4d4a962b94f174658acb WatchSource:0}: Error finding container 7ca4366b4f0438928f20f36039d2d58c8406608d147c4d4a962b94f174658acb: Status 404 returned error can't find the container with id 7ca4366b4f0438928f20f36039d2d58c8406608d147c4d4a962b94f174658acb Nov 01 16:15:17.512114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.511927 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6"] Nov 01 16:15:17.615551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.615511 4911 generic.go:356] "Generic (PLEG): container finished" podID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerID="2b3205fa5a0bc9e6b831e4ab00486a3c71f8fcf64f16ecee5a2e2937d8c397d7" exitCode=60 Nov 01 16:15:17.615988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.615580 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerDied","Data":"2b3205fa5a0bc9e6b831e4ab00486a3c71f8fcf64f16ecee5a2e2937d8c397d7"} Nov 01 16:15:17.615988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.615643 4911 scope.go:122] "RemoveContainer" containerID="eb521b003f6603c368e2025bfa888d8d915ca0e6a4c2589ca5c874e232c686c9" Nov 01 16:15:17.616172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.616145 4911 scope.go:122] "RemoveContainer" containerID="2b3205fa5a0bc9e6b831e4ab00486a3c71f8fcf64f16ecee5a2e2937d8c397d7" Nov 01 16:15:17.616953 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:17.616649 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc_odh-ai-gateway-infra(9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" Nov 01 16:15:17.617570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.617540 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" Nov 01 16:15:17.617570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.617553 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr" event={"ID":"1d811349-4144-4d51-a15e-5f81feb6b7ed","Type":"ContainerDied","Data":"8a3bcd5490051dc347f25db982e06cf36ce239eebdd09b0bb8c12596608914a7"} Nov 01 16:15:17.619787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.619755 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" event={"ID":"fa496de7-fb8d-4329-9302-b6983e105838","Type":"ContainerStarted","Data":"7ca4366b4f0438928f20f36039d2d58c8406608d147c4d4a962b94f174658acb"} Nov 01 16:15:17.635951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.635930 4911 scope.go:122] "RemoveContainer" containerID="2a076241f69d9de1d76fec0c14c2d7fbbf4fa0b3adc741887072ae8e8f3bd81d" Nov 01 16:15:17.664199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.664165 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc"] Nov 01 16:15:17.669721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.669693 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr"] Nov 01 16:15:17.680951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.680923 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892495-n42gr"] Nov 01 16:15:17.979816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:17.979782 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" path="/var/lib/kubelet/pods/1d811349-4144-4d51-a15e-5f81feb6b7ed/volumes" Nov 01 16:15:18.645229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.645186 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" event={"ID":"fa496de7-fb8d-4329-9302-b6983e105838","Type":"ContainerStarted","Data":"e2400e450f4ca3fd6751735e0acd210c2860dbbb9649231c4f4a406eed475fbe"} Nov 01 16:15:18.666092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.666010 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" podStartSLOduration=18.665991799 podStartE2EDuration="18.665991799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:18.66145771 +0000 UTC m=+1658.805912353" watchObservedRunningTime="2026-11-01 16:15:18.665991799 +0000 UTC m=+1658.810446444" Nov 01 16:15:18.901998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.901934 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:18.938969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.938933 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft"] Nov 01 16:15:18.940035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940008 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940031 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940070 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940078 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940104 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940110 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940295 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940307 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:18.940377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.940327 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d811349-4144-4d51-a15e-5f81feb6b7ed" containerName="cleanup" Nov 01 16:15:18.950653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.950603 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:18.956283 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:18.956239 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft"] Nov 01 16:15:19.018402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.018368 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-maas-api-service-ca\") pod \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " Nov 01 16:15:19.018589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.018502 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-kube-api-access-mb9ls\" (UniqueName: \"kubernetes.io/projected/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-kube-api-access-mb9ls\") pod \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\" (UID: \"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2\") " Nov 01 16:15:19.018766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.018731 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-maas-api-service-ca" pod "9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" (UID: "9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:19.019658 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.019636 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:19.021432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.021409 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-kube-api-access-mb9ls" pod "9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" (UID: "9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2"). InnerVolumeSpecName "kube-api-access-mb9ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:19.121496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.121450 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xscnk\" (UniqueName: \"kubernetes.io/projected/b292b099-0e0e-4f27-8fb7-db1a2e22e768-kube-api-access-xscnk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:19.122649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.122443 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b292b099-0e0e-4f27-8fb7-db1a2e22e768-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:19.122816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.122710 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9ls\" (UniqueName: \"kubernetes.io/projected/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2-kube-api-access-mb9ls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:19.224300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.224215 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b292b099-0e0e-4f27-8fb7-db1a2e22e768-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:19.224300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.224268 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xscnk\" (UniqueName: \"kubernetes.io/projected/b292b099-0e0e-4f27-8fb7-db1a2e22e768-kube-api-access-xscnk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:19.224864 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.224843 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b292b099-0e0e-4f27-8fb7-db1a2e22e768-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:19.233480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.233453 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xscnk\" (UniqueName: \"kubernetes.io/projected/b292b099-0e0e-4f27-8fb7-db1a2e22e768-kube-api-access-xscnk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:19.281402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.281369 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:19.585523 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:19.585481 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb292b099_0e0e_4f27_8fb7_db1a2e22e768.slice/crio-7ef2d15de5311bffc88e32dfc1f92e459d0d0c4529022f12fdf9b0449072cf2b WatchSource:0}: Error finding container 7ef2d15de5311bffc88e32dfc1f92e459d0d0c4529022f12fdf9b0449072cf2b: Status 404 returned error can't find the container with id 7ef2d15de5311bffc88e32dfc1f92e459d0d0c4529022f12fdf9b0449072cf2b Nov 01 16:15:19.586883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.586857 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft"] Nov 01 16:15:19.659128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.659095 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" Nov 01 16:15:19.659553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.659098 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc" event={"ID":"9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2","Type":"ContainerDied","Data":"a64cccb7d667a93bebee0b954f85af716ee1cd32cd0cb59e13729b3ec210593c"} Nov 01 16:15:19.659553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.659328 4911 scope.go:122] "RemoveContainer" containerID="2b3205fa5a0bc9e6b831e4ab00486a3c71f8fcf64f16ecee5a2e2937d8c397d7" Nov 01 16:15:19.661855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.661819 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" event={"ID":"b292b099-0e0e-4f27-8fb7-db1a2e22e768","Type":"ContainerStarted","Data":"7ef2d15de5311bffc88e32dfc1f92e459d0d0c4529022f12fdf9b0449072cf2b"} Nov 01 16:15:19.663939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.663912 4911 generic.go:356] "Generic (PLEG): container finished" podID="fa496de7-fb8d-4329-9302-b6983e105838" containerID="e2400e450f4ca3fd6751735e0acd210c2860dbbb9649231c4f4a406eed475fbe" exitCode=60 Nov 01 16:15:19.664101 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.663976 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" event={"ID":"fa496de7-fb8d-4329-9302-b6983e105838","Type":"ContainerDied","Data":"e2400e450f4ca3fd6751735e0acd210c2860dbbb9649231c4f4a406eed475fbe"} Nov 01 16:15:19.664533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.664513 4911 scope.go:122] "RemoveContainer" containerID="e2400e450f4ca3fd6751735e0acd210c2860dbbb9649231c4f4a406eed475fbe" Nov 01 16:15:19.703618 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.703575 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc"] Nov 01 16:15:19.712169 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.712141 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892495-442gc"] Nov 01 16:15:19.971429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:19.971390 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" path="/var/lib/kubelet/pods/9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2/volumes" Nov 01 16:15:20.675515 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:20.675475 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" event={"ID":"b292b099-0e0e-4f27-8fb7-db1a2e22e768","Type":"ContainerStarted","Data":"8f4ef331f2d52db8b1fcedcd9d5ef917ae9ca7e70798c8be49eb7632fda06501"} Nov 01 16:15:20.677808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:20.677770 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" event={"ID":"fa496de7-fb8d-4329-9302-b6983e105838","Type":"ContainerStarted","Data":"42ebd1de6fa65a12e083c9954cbbc554c9fb027e349d8bf28f39d3817215d0ca"} Nov 01 16:15:20.693363 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:20.693316 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" podStartSLOduration=20.693302557 podStartE2EDuration="20.693302557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:20.69043033 +0000 UTC m=+1660.834884974" watchObservedRunningTime="2026-11-01 16:15:20.693302557 +0000 UTC m=+1660.837757199" Nov 01 16:15:21.693082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.693049 4911 generic.go:356] "Generic (PLEG): container finished" podID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerID="8f4ef331f2d52db8b1fcedcd9d5ef917ae9ca7e70798c8be49eb7632fda06501" exitCode=60 Nov 01 16:15:21.693537 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.693095 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" event={"ID":"b292b099-0e0e-4f27-8fb7-db1a2e22e768","Type":"ContainerDied","Data":"8f4ef331f2d52db8b1fcedcd9d5ef917ae9ca7e70798c8be49eb7632fda06501"} Nov 01 16:15:21.693537 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.693485 4911 scope.go:122] "RemoveContainer" containerID="8f4ef331f2d52db8b1fcedcd9d5ef917ae9ca7e70798c8be49eb7632fda06501" Nov 01 16:15:21.695598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.695486 4911 generic.go:356] "Generic (PLEG): container finished" podID="fa496de7-fb8d-4329-9302-b6983e105838" containerID="42ebd1de6fa65a12e083c9954cbbc554c9fb027e349d8bf28f39d3817215d0ca" exitCode=60 Nov 01 16:15:21.695598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.695562 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" event={"ID":"fa496de7-fb8d-4329-9302-b6983e105838","Type":"ContainerDied","Data":"42ebd1de6fa65a12e083c9954cbbc554c9fb027e349d8bf28f39d3817215d0ca"} Nov 01 16:15:21.695689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.695612 4911 scope.go:122] "RemoveContainer" containerID="e2400e450f4ca3fd6751735e0acd210c2860dbbb9649231c4f4a406eed475fbe" Nov 01 16:15:21.695883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.695863 4911 scope.go:122] "RemoveContainer" containerID="42ebd1de6fa65a12e083c9954cbbc554c9fb027e349d8bf28f39d3817215d0ca" Nov 01 16:15:21.696241 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:21.696216 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6_odh-ai-gateway-infra(fa496de7-fb8d-4329-9302-b6983e105838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" podUID="fa496de7-fb8d-4329-9302-b6983e105838" Nov 01 16:15:21.960168 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:21.959690 4911 scope.go:122] "RemoveContainer" containerID="377d9c8c9ee70fd8bce0b395f82fc3bce70cfd4858ce54bbed5f08beac38ebe5" Nov 01 16:15:22.707271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:22.707227 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerStarted","Data":"a06f63b322d6142152a0fc459525725f7027847014d8d9377d9ed9be6818314d"} Nov 01 16:15:22.709756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:22.709727 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" event={"ID":"b292b099-0e0e-4f27-8fb7-db1a2e22e768","Type":"ContainerStarted","Data":"01a705d7773e1887bc01f4541f74128ccd6d79a73dc36830467abb20ba234f1c"} Nov 01 16:15:22.712622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:22.712588 4911 scope.go:122] "RemoveContainer" containerID="42ebd1de6fa65a12e083c9954cbbc554c9fb027e349d8bf28f39d3817215d0ca" Nov 01 16:15:22.712993 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:22.712971 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6_odh-ai-gateway-infra(fa496de7-fb8d-4329-9302-b6983e105838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" podUID="fa496de7-fb8d-4329-9302-b6983e105838" Nov 01 16:15:22.959420 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:22.959369 4911 scope.go:122] "RemoveContainer" containerID="ac491dde7bb3c22c12892dc8a659c955bcefe8c941c83987e8bbeeca500fc354" Nov 01 16:15:22.959833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:22.959801 4911 scope.go:122] "RemoveContainer" containerID="73cbce92fa987abcd70c82ccf17bbae6e50fbb371879ba9000fe0b6972d752c2" Nov 01 16:15:23.724088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.723981 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerStarted","Data":"ec7b1e74ab8943d059e1bd1a7017cfdd5e629eaf81a76bdc2a19ee7742c05121"} Nov 01 16:15:23.726499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.726472 4911 generic.go:356] "Generic (PLEG): container finished" podID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerID="01a705d7773e1887bc01f4541f74128ccd6d79a73dc36830467abb20ba234f1c" exitCode=60 Nov 01 16:15:23.726623 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.726552 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" event={"ID":"b292b099-0e0e-4f27-8fb7-db1a2e22e768","Type":"ContainerDied","Data":"01a705d7773e1887bc01f4541f74128ccd6d79a73dc36830467abb20ba234f1c"} Nov 01 16:15:23.726623 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.726615 4911 scope.go:122] "RemoveContainer" containerID="8f4ef331f2d52db8b1fcedcd9d5ef917ae9ca7e70798c8be49eb7632fda06501" Nov 01 16:15:23.726889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.726867 4911 scope.go:122] "RemoveContainer" containerID="01a705d7773e1887bc01f4541f74128ccd6d79a73dc36830467abb20ba234f1c" Nov 01 16:15:23.727389 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:23.727360 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft_odh-ai-gateway-infra(b292b099-0e0e-4f27-8fb7-db1a2e22e768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" Nov 01 16:15:23.729756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.729734 4911 generic.go:356] "Generic (PLEG): container finished" podID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerID="a06f63b322d6142152a0fc459525725f7027847014d8d9377d9ed9be6818314d" exitCode=60 Nov 01 16:15:23.729858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.729804 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerDied","Data":"a06f63b322d6142152a0fc459525725f7027847014d8d9377d9ed9be6818314d"} Nov 01 16:15:23.745088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.730430 4911 scope.go:122] "RemoveContainer" containerID="a06f63b322d6142152a0fc459525725f7027847014d8d9377d9ed9be6818314d" Nov 01 16:15:23.745088 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:23.730845 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm_odh-ai-gateway-infra(dfda58ef-e7b3-47cb-b357-ba60d84a04cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" Nov 01 16:15:23.745088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.732738 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerStarted","Data":"31d66231bb1e7f2d8c12b01b15f5dad1fd3fa379ea609dfdf61058a00d3493dd"} Nov 01 16:15:23.746578 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.746556 4911 scope.go:122] "RemoveContainer" containerID="377d9c8c9ee70fd8bce0b395f82fc3bce70cfd4858ce54bbed5f08beac38ebe5" Nov 01 16:15:23.746681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.746658 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm"] Nov 01 16:15:23.978759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.978671 4911 scope.go:122] "RemoveContainer" containerID="d940c711a6cc518dbe236304dbe2378fc3f88d50c4332a62b122f233baa7f49d" Nov 01 16:15:23.978940 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.978921 4911 scope.go:122] "RemoveContainer" containerID="2317819da2a7530b7d551802d60e8f18b04dbbcdd64649b85869e860db4f0610" Nov 01 16:15:23.979155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:23.979132 4911 scope.go:122] "RemoveContainer" containerID="e5641ed028844d5a25264fbbeafea7c264b79a0178b45d58d384761045027110" Nov 01 16:15:24.744250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.744209 4911 scope.go:122] "RemoveContainer" containerID="01a705d7773e1887bc01f4541f74128ccd6d79a73dc36830467abb20ba234f1c" Nov 01 16:15:24.744705 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:24.744678 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft_odh-ai-gateway-infra(b292b099-0e0e-4f27-8fb7-db1a2e22e768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" Nov 01 16:15:24.746170 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.746138 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerStarted","Data":"1fb36afdb00a201d1ff211c292db559e7dc25e885c6eb883cafb2eb82bcb8603"} Nov 01 16:15:24.752295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.752267 4911 generic.go:356] "Generic (PLEG): container finished" podID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerID="31d66231bb1e7f2d8c12b01b15f5dad1fd3fa379ea609dfdf61058a00d3493dd" exitCode=60 Nov 01 16:15:24.752416 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.752341 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerDied","Data":"31d66231bb1e7f2d8c12b01b15f5dad1fd3fa379ea609dfdf61058a00d3493dd"} Nov 01 16:15:24.752481 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.752404 4911 scope.go:122] "RemoveContainer" containerID="73cbce92fa987abcd70c82ccf17bbae6e50fbb371879ba9000fe0b6972d752c2" Nov 01 16:15:24.754088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.752771 4911 scope.go:122] "RemoveContainer" containerID="31d66231bb1e7f2d8c12b01b15f5dad1fd3fa379ea609dfdf61058a00d3493dd" Nov 01 16:15:24.754088 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:24.753177 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6_odh-ai-gateway-infra(0324204c-7d4f-4644-aa70-39e7e8a0e7e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" Nov 01 16:15:24.768133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.768101 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp"] Nov 01 16:15:24.782962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.782923 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" event={"ID":"9eef4f46-6006-4038-8209-77ed34168412","Type":"ContainerStarted","Data":"ad149fe3fd8a99fdffa17ebd15d1613d77bcb86930eaaae06d8a64193c284139"} Nov 01 16:15:24.790847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.790812 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" event={"ID":"d5321715-5daf-4973-9218-0ce2ac3a0282","Type":"ContainerStarted","Data":"acdcfa91b32500fe4c7e61a7963228c2016728cd191324dbcd783a348f7adb5d"} Nov 01 16:15:24.793992 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.793963 4911 generic.go:356] "Generic (PLEG): container finished" podID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerID="ec7b1e74ab8943d059e1bd1a7017cfdd5e629eaf81a76bdc2a19ee7742c05121" exitCode=60 Nov 01 16:15:24.794173 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.794026 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerDied","Data":"ec7b1e74ab8943d059e1bd1a7017cfdd5e629eaf81a76bdc2a19ee7742c05121"} Nov 01 16:15:24.798625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.798607 4911 scope.go:122] "RemoveContainer" containerID="ac491dde7bb3c22c12892dc8a659c955bcefe8c941c83987e8bbeeca500fc354" Nov 01 16:15:24.825547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.825514 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6"] Nov 01 16:15:24.968097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.960436 4911 scope.go:122] "RemoveContainer" containerID="ebe46f8b9cae74ba05eafd54fbd88484543f7eb4f9481c402ac1a9261ead3a60" Nov 01 16:15:24.968097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.963897 4911 scope.go:122] "RemoveContainer" containerID="f84c12d7e673113bec31a1380024b1b7cefb894d2d2c17f37bf4bb44e3bf8525" Nov 01 16:15:24.968097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:24.964439 4911 scope.go:122] "RemoveContainer" containerID="6f6f5683591e6045d45eb0550505c727b5f2b2f03e09cf23453ad5d8210b69fd" Nov 01 16:15:25.083334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.083305 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:25.169271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.169237 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-kube-api-access-tnb8r\" (UniqueName: \"kubernetes.io/projected/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-kube-api-access-tnb8r\") pod \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " Nov 01 16:15:25.169435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.169339 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-maas-api-service-ca\") pod \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\" (UID: \"dfda58ef-e7b3-47cb-b357-ba60d84a04cc\") " Nov 01 16:15:25.170690 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.170663 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-maas-api-service-ca" pod "dfda58ef-e7b3-47cb-b357-ba60d84a04cc" (UID: "dfda58ef-e7b3-47cb-b357-ba60d84a04cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:25.172734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.172708 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-kube-api-access-tnb8r" pod "dfda58ef-e7b3-47cb-b357-ba60d84a04cc" (UID: "dfda58ef-e7b3-47cb-b357-ba60d84a04cc"). InnerVolumeSpecName "kube-api-access-tnb8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:25.238750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.238724 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:25.271919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.271889 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-maas-api-service-ca\") pod \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " Nov 01 16:15:25.272227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.272205 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-kube-api-access-xmfhc\" (UniqueName: \"kubernetes.io/projected/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-kube-api-access-xmfhc\") pod \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\" (UID: \"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8\") " Nov 01 16:15:25.272324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.272305 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-maas-api-service-ca" pod "d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" (UID: "d2ba2926-33bf-4a15-b31d-f40ddd6c72c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:25.273119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.273096 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnb8r\" (UniqueName: \"kubernetes.io/projected/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-kube-api-access-tnb8r\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:25.273119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.273121 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfda58ef-e7b3-47cb-b357-ba60d84a04cc-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:25.273256 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.273131 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:25.274428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.274407 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-kube-api-access-xmfhc" pod "d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" (UID: "d2ba2926-33bf-4a15-b31d-f40ddd6c72c8"). InnerVolumeSpecName "kube-api-access-xmfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:25.374641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.374606 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfhc\" (UniqueName: \"kubernetes.io/projected/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8-kube-api-access-xmfhc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:25.802021 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.801976 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4"] Nov 01 16:15:25.811977 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.811931 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" event={"ID":"7f085329-c40e-48bb-9978-fe2f8fb00695","Type":"ContainerStarted","Data":"4eb4eb2953a701a497796e3e4363ca475782fe61ff14d6839db0fabc22f46b7b"} Nov 01 16:15:25.815765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.815731 4911 generic.go:356] "Generic (PLEG): container finished" podID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerID="1fb36afdb00a201d1ff211c292db559e7dc25e885c6eb883cafb2eb82bcb8603" exitCode=60 Nov 01 16:15:25.815907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.815813 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerDied","Data":"1fb36afdb00a201d1ff211c292db559e7dc25e885c6eb883cafb2eb82bcb8603"} Nov 01 16:15:25.815907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.815877 4911 scope.go:122] "RemoveContainer" containerID="2317819da2a7530b7d551802d60e8f18b04dbbcdd64649b85869e860db4f0610" Nov 01 16:15:25.821091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.817860 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" Nov 01 16:15:25.821091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.817875 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm" event={"ID":"dfda58ef-e7b3-47cb-b357-ba60d84a04cc","Type":"ContainerDied","Data":"ae05f064a9d6444fc634a1c2cc6f8b32db0256b1219f78574f9474fc98a6961e"} Nov 01 16:15:25.821091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.820334 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerStarted","Data":"3b7d9328edf5041d95fe8824b1e5ce6b3c2ebe31ae6a0f02cf43d315360d37e9"} Nov 01 16:15:25.828056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.828021 4911 generic.go:356] "Generic (PLEG): container finished" podID="9eef4f46-6006-4038-8209-77ed34168412" containerID="ad149fe3fd8a99fdffa17ebd15d1613d77bcb86930eaaae06d8a64193c284139" exitCode=60 Nov 01 16:15:25.828278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.828255 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" event={"ID":"9eef4f46-6006-4038-8209-77ed34168412","Type":"ContainerDied","Data":"ad149fe3fd8a99fdffa17ebd15d1613d77bcb86930eaaae06d8a64193c284139"} Nov 01 16:15:25.828988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.828969 4911 scope.go:122] "RemoveContainer" containerID="ad149fe3fd8a99fdffa17ebd15d1613d77bcb86930eaaae06d8a64193c284139" Nov 01 16:15:25.829799 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:25.829774 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht_odh-ai-gateway-infra(9eef4f46-6006-4038-8209-77ed34168412)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" podUID="9eef4f46-6006-4038-8209-77ed34168412" Nov 01 16:15:25.836096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.831432 4911 generic.go:356] "Generic (PLEG): container finished" podID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerID="acdcfa91b32500fe4c7e61a7963228c2016728cd191324dbcd783a348f7adb5d" exitCode=60 Nov 01 16:15:25.836096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.831519 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" event={"ID":"d5321715-5daf-4973-9218-0ce2ac3a0282","Type":"ContainerDied","Data":"acdcfa91b32500fe4c7e61a7963228c2016728cd191324dbcd783a348f7adb5d"} Nov 01 16:15:25.836096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.831948 4911 scope.go:122] "RemoveContainer" containerID="acdcfa91b32500fe4c7e61a7963228c2016728cd191324dbcd783a348f7adb5d" Nov 01 16:15:25.836096 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:25.832368 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp_odh-ai-gateway-infra(d5321715-5daf-4973-9218-0ce2ac3a0282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" Nov 01 16:15:25.838648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.838620 4911 scope.go:122] "RemoveContainer" containerID="a06f63b322d6142152a0fc459525725f7027847014d8d9377d9ed9be6818314d" Nov 01 16:15:25.841195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.841167 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" event={"ID":"d2ba2926-33bf-4a15-b31d-f40ddd6c72c8","Type":"ContainerDied","Data":"32433749eaf1f995d70088df1a10abf787f10073fd042212ea14e13c6ea25376"} Nov 01 16:15:25.841293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.841181 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp" Nov 01 16:15:25.845392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.845359 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerStarted","Data":"f8bf14b76dd03afc0f182eaca287c0794126c17aa50c04dc1b8ce924d0fc91ea"} Nov 01 16:15:25.862287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.862261 4911 scope.go:122] "RemoveContainer" containerID="d940c711a6cc518dbe236304dbe2378fc3f88d50c4332a62b122f233baa7f49d" Nov 01 16:15:25.883694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.883669 4911 scope.go:122] "RemoveContainer" containerID="e5641ed028844d5a25264fbbeafea7c264b79a0178b45d58d384761045027110" Nov 01 16:15:25.897361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.897326 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm"] Nov 01 16:15:25.906567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.906515 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht"] Nov 01 16:15:25.932653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.919701 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1c65e3-29892495-bz2bm"] Nov 01 16:15:25.941363 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:25.941331 4911 scope.go:122] "RemoveContainer" containerID="ec7b1e74ab8943d059e1bd1a7017cfdd5e629eaf81a76bdc2a19ee7742c05121" Nov 01 16:15:26.005573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.004866 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" path="/var/lib/kubelet/pods/dfda58ef-e7b3-47cb-b357-ba60d84a04cc/volumes" Nov 01 16:15:26.005573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.005543 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp"] Nov 01 16:15:26.006265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.006239 4911 scope.go:122] "RemoveContainer" containerID="a49e819a1307a3476a426a302cbd4b54c6dc341ba0a265bab2e5e981d4d52596" Nov 01 16:15:26.062305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.062090 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp"] Nov 01 16:15:26.077058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.077018 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-375350-29892495-srzjp"] Nov 01 16:15:26.117717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.117687 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:26.154500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.154454 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6"] Nov 01 16:15:26.157609 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.157581 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:26.157755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.157741 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:26.157898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.157886 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.157988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.157978 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.158099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158088 4911 cpu_manager.go:414] "containerMap: removing container" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.158179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158169 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.158243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158235 4911 cpu_manager.go:414] "containerMap: removing container" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.158314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158305 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.158386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158376 4911 cpu_manager.go:414] "containerMap: removing container" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.158457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158448 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.158521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158512 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:26.158577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158569 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:26.158661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158653 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.158786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158775 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.158854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158846 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.158914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.158905 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.159230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159212 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.159346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159335 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:26.159430 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159420 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd22252-dc4e-41bd-8dfe-2e0dc3fb4ec2" containerName="cleanup" Nov 01 16:15:26.159511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159502 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:26.159598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159589 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.159661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159653 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.159724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159715 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfda58ef-e7b3-47cb-b357-ba60d84a04cc" containerName="cleanup" Nov 01 16:15:26.159800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.159791 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:26.196756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.196656 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7be1a72-b700-49aa-8b9c-d80cdc39168a-kube-api-access-b4vvf\" (UniqueName: \"kubernetes.io/projected/a7be1a72-b700-49aa-8b9c-d80cdc39168a-kube-api-access-b4vvf\") pod \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " Nov 01 16:15:26.197247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.197220 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7be1a72-b700-49aa-8b9c-d80cdc39168a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7be1a72-b700-49aa-8b9c-d80cdc39168a-maas-api-service-ca\") pod \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\" (UID: \"a7be1a72-b700-49aa-8b9c-d80cdc39168a\") " Nov 01 16:15:26.198178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.198150 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.198178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.198167 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7be1a72-b700-49aa-8b9c-d80cdc39168a-maas-api-service-ca" pod "a7be1a72-b700-49aa-8b9c-d80cdc39168a" (UID: "a7be1a72-b700-49aa-8b9c-d80cdc39168a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:26.198738 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.198716 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7be1a72-b700-49aa-8b9c-d80cdc39168a-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:26.202602 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.202576 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6"] Nov 01 16:15:26.204456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.204436 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0c091d-dockercfg-w9frs\"" Nov 01 16:15:26.204631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.204619 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0c091d\"" Nov 01 16:15:26.252417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.252211 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7be1a72-b700-49aa-8b9c-d80cdc39168a-kube-api-access-b4vvf" pod "a7be1a72-b700-49aa-8b9c-d80cdc39168a" (UID: "a7be1a72-b700-49aa-8b9c-d80cdc39168a"). InnerVolumeSpecName "kube-api-access-b4vvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:26.313542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.301290 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2db7\" (UniqueName: \"kubernetes.io/projected/3eb9a3c2-532a-45f4-9489-b147b2969fab-kube-api-access-m2db7\") pod \"payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.313542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.301484 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9a3c2-532a-45f4-9489-b147b2969fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.313542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.301864 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4vvf\" (UniqueName: \"kubernetes.io/projected/a7be1a72-b700-49aa-8b9c-d80cdc39168a-kube-api-access-b4vvf\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:26.336836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.336803 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:26.403298 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.403262 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-maas-api-service-ca\") pod \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " Nov 01 16:15:26.403465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.403312 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-kube-api-access-dlxj7\" (UniqueName: \"kubernetes.io/projected/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-kube-api-access-dlxj7\") pod \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\" (UID: \"0324204c-7d4f-4644-aa70-39e7e8a0e7e7\") " Nov 01 16:15:26.403712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.403683 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-maas-api-service-ca" pod "0324204c-7d4f-4644-aa70-39e7e8a0e7e7" (UID: "0324204c-7d4f-4644-aa70-39e7e8a0e7e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:26.404346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.404305 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2db7\" (UniqueName: \"kubernetes.io/projected/3eb9a3c2-532a-45f4-9489-b147b2969fab-kube-api-access-m2db7\") pod \"payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.404550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.404523 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9a3c2-532a-45f4-9489-b147b2969fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.405326 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.404648 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:26.405619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.405590 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-kube-api-access-dlxj7" pod "0324204c-7d4f-4644-aa70-39e7e8a0e7e7" (UID: "0324204c-7d4f-4644-aa70-39e7e8a0e7e7"). InnerVolumeSpecName "kube-api-access-dlxj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:26.405697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.405670 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9a3c2-532a-45f4-9489-b147b2969fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.412887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.412862 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2db7\" (UniqueName: \"kubernetes.io/projected/3eb9a3c2-532a-45f4-9489-b147b2969fab-kube-api-access-m2db7\") pod \"payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.506535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.506502 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxj7\" (UniqueName: \"kubernetes.io/projected/0324204c-7d4f-4644-aa70-39e7e8a0e7e7-kube-api-access-dlxj7\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:26.681597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.681564 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:26.858852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.858814 4911 generic.go:356] "Generic (PLEG): container finished" podID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerID="4eb4eb2953a701a497796e3e4363ca475782fe61ff14d6839db0fabc22f46b7b" exitCode=60 Nov 01 16:15:26.859341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.858891 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" event={"ID":"7f085329-c40e-48bb-9978-fe2f8fb00695","Type":"ContainerDied","Data":"4eb4eb2953a701a497796e3e4363ca475782fe61ff14d6839db0fabc22f46b7b"} Nov 01 16:15:26.859341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.858963 4911 scope.go:122] "RemoveContainer" containerID="f84c12d7e673113bec31a1380024b1b7cefb894d2d2c17f37bf4bb44e3bf8525" Nov 01 16:15:26.859613 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.859484 4911 scope.go:122] "RemoveContainer" containerID="4eb4eb2953a701a497796e3e4363ca475782fe61ff14d6839db0fabc22f46b7b" Nov 01 16:15:26.860017 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:26.859993 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs_odh-ai-gateway-infra(7f085329-c40e-48bb-9978-fe2f8fb00695)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" Nov 01 16:15:26.861607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.861586 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs"] Nov 01 16:15:26.862342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.862322 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" Nov 01 16:15:26.862683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.862651 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4" event={"ID":"a7be1a72-b700-49aa-8b9c-d80cdc39168a","Type":"ContainerDied","Data":"3c2bc1f80570ff45244fab8d438b9d714ff088d7fbb63d19c817485b0eb92913"} Nov 01 16:15:26.866845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.866822 4911 generic.go:356] "Generic (PLEG): container finished" podID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerID="3b7d9328edf5041d95fe8824b1e5ce6b3c2ebe31ae6a0f02cf43d315360d37e9" exitCode=60 Nov 01 16:15:26.866922 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.866902 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerDied","Data":"3b7d9328edf5041d95fe8824b1e5ce6b3c2ebe31ae6a0f02cf43d315360d37e9"} Nov 01 16:15:26.867456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.867433 4911 scope.go:122] "RemoveContainer" containerID="3b7d9328edf5041d95fe8824b1e5ce6b3c2ebe31ae6a0f02cf43d315360d37e9" Nov 01 16:15:26.867861 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:26.867834 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll_odh-ai-gateway-infra(ed858f9a-835c-4c94-9866-5f28cdf2550b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" Nov 01 16:15:26.869717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.869694 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" Nov 01 16:15:26.869822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.869727 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6" event={"ID":"0324204c-7d4f-4644-aa70-39e7e8a0e7e7","Type":"ContainerDied","Data":"8e50bd155c626db33558c5015410b5053f70073350f9b2a1b235fc5d6f0eed53"} Nov 01 16:15:26.880877 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.880847 4911 scope.go:122] "RemoveContainer" containerID="1fb36afdb00a201d1ff211c292db559e7dc25e885c6eb883cafb2eb82bcb8603" Nov 01 16:15:26.886720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.886697 4911 generic.go:356] "Generic (PLEG): container finished" podID="422af011-c613-4443-8047-d86cb44cd89f" containerID="f8bf14b76dd03afc0f182eaca287c0794126c17aa50c04dc1b8ce924d0fc91ea" exitCode=60 Nov 01 16:15:26.886798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.886767 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerDied","Data":"f8bf14b76dd03afc0f182eaca287c0794126c17aa50c04dc1b8ce924d0fc91ea"} Nov 01 16:15:26.887437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.887322 4911 scope.go:122] "RemoveContainer" containerID="f8bf14b76dd03afc0f182eaca287c0794126c17aa50c04dc1b8ce924d0fc91ea" Nov 01 16:15:26.887789 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:26.887767 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z_odh-ai-gateway-infra(422af011-c613-4443-8047-d86cb44cd89f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" podUID="422af011-c613-4443-8047-d86cb44cd89f" Nov 01 16:15:26.889314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.889294 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerStarted","Data":"56adb93134d6f9d66cb9c5c4f664bbc9dba6c627f24365d72f8ac40fccc87bed"} Nov 01 16:15:26.896712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.896689 4911 scope.go:122] "RemoveContainer" containerID="ebe46f8b9cae74ba05eafd54fbd88484543f7eb4f9481c402ac1a9261ead3a60" Nov 01 16:15:26.917231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.916516 4911 scope.go:122] "RemoveContainer" containerID="31d66231bb1e7f2d8c12b01b15f5dad1fd3fa379ea609dfdf61058a00d3493dd" Nov 01 16:15:26.931211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.931185 4911 scope.go:122] "RemoveContainer" containerID="6f6f5683591e6045d45eb0550505c727b5f2b2f03e09cf23453ad5d8210b69fd" Nov 01 16:15:26.931800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.931771 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6"] Nov 01 16:15:26.993333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:26.993185 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll"] Nov 01 16:15:27.001246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.001215 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892495-9vmj6"] Nov 01 16:15:27.064794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.064758 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6"] Nov 01 16:15:27.065700 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:15:27.065640 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb9a3c2_532a_45f4_9489_b147b2969fab.slice/crio-bfd6f03b98a1321284248cd03392cfa75b9a006b81454593383631fdab8babbf WatchSource:0}: Error finding container bfd6f03b98a1321284248cd03392cfa75b9a006b81454593383631fdab8babbf: Status 404 returned error can't find the container with id bfd6f03b98a1321284248cd03392cfa75b9a006b81454593383631fdab8babbf Nov 01 16:15:27.091140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.090736 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z"] Nov 01 16:15:27.154798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.154771 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4"] Nov 01 16:15:27.201682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.201647 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892495-vrqr4"] Nov 01 16:15:27.231060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.218243 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:27.334070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.324288 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eef4f46-6006-4038-8209-77ed34168412-kube-api-access-t4mnp\" (UniqueName: \"kubernetes.io/projected/9eef4f46-6006-4038-8209-77ed34168412-kube-api-access-t4mnp\") pod \"9eef4f46-6006-4038-8209-77ed34168412\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " Nov 01 16:15:27.334070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.324474 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eef4f46-6006-4038-8209-77ed34168412-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eef4f46-6006-4038-8209-77ed34168412-maas-api-service-ca\") pod \"9eef4f46-6006-4038-8209-77ed34168412\" (UID: \"9eef4f46-6006-4038-8209-77ed34168412\") " Nov 01 16:15:27.334070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.325831 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eef4f46-6006-4038-8209-77ed34168412-maas-api-service-ca" pod "9eef4f46-6006-4038-8209-77ed34168412" (UID: "9eef4f46-6006-4038-8209-77ed34168412"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:27.334070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.328954 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eef4f46-6006-4038-8209-77ed34168412-kube-api-access-t4mnp" pod "9eef4f46-6006-4038-8209-77ed34168412" (UID: "9eef4f46-6006-4038-8209-77ed34168412"). InnerVolumeSpecName "kube-api-access-t4mnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:27.451222 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.451192 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:27.582586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.580077 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eef4f46-6006-4038-8209-77ed34168412-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:27.582839 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.582814 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mnp\" (UniqueName: \"kubernetes.io/projected/9eef4f46-6006-4038-8209-77ed34168412-kube-api-access-t4mnp\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:27.692064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.685624 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5321715-5daf-4973-9218-0ce2ac3a0282-kube-api-access-tqsls\" (UniqueName: \"kubernetes.io/projected/d5321715-5daf-4973-9218-0ce2ac3a0282-kube-api-access-tqsls\") pod \"d5321715-5daf-4973-9218-0ce2ac3a0282\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " Nov 01 16:15:27.692064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.685744 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5321715-5daf-4973-9218-0ce2ac3a0282-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5321715-5daf-4973-9218-0ce2ac3a0282-maas-api-service-ca\") pod \"d5321715-5daf-4973-9218-0ce2ac3a0282\" (UID: \"d5321715-5daf-4973-9218-0ce2ac3a0282\") " Nov 01 16:15:27.692064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.689389 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5321715-5daf-4973-9218-0ce2ac3a0282-maas-api-service-ca" pod "d5321715-5daf-4973-9218-0ce2ac3a0282" (UID: "d5321715-5daf-4973-9218-0ce2ac3a0282"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:27.738070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.737267 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5321715-5daf-4973-9218-0ce2ac3a0282-kube-api-access-tqsls" pod "d5321715-5daf-4973-9218-0ce2ac3a0282" (UID: "d5321715-5daf-4973-9218-0ce2ac3a0282"). InnerVolumeSpecName "kube-api-access-tqsls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:27.790862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.790831 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqsls\" (UniqueName: \"kubernetes.io/projected/d5321715-5daf-4973-9218-0ce2ac3a0282-kube-api-access-tqsls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:27.791029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:27.791016 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5321715-5daf-4973-9218-0ce2ac3a0282-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.032079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.029289 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" Nov 01 16:15:28.040060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.037731 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" Nov 01 16:15:28.074788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.073727 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" path="/var/lib/kubelet/pods/0324204c-7d4f-4644-aa70-39e7e8a0e7e7/volumes" Nov 01 16:15:28.074788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.074558 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" path="/var/lib/kubelet/pods/a7be1a72-b700-49aa-8b9c-d80cdc39168a/volumes" Nov 01 16:15:28.075427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.075371 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" path="/var/lib/kubelet/pods/d2ba2926-33bf-4a15-b31d-f40ddd6c72c8/volumes" Nov 01 16:15:28.092412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.076939 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht" event={"ID":"9eef4f46-6006-4038-8209-77ed34168412","Type":"ContainerDied","Data":"28e444890d59ac8046f52e1308e52d541d84d936d2df184911e9667b0713fab0"} Nov 01 16:15:28.092412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.076981 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp" event={"ID":"d5321715-5daf-4973-9218-0ce2ac3a0282","Type":"ContainerDied","Data":"c44f487bc46c486e0c18e7b4bb5f8e0464660a48a3c80ccc6be36ea430fc14df"} Nov 01 16:15:28.092412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.077018 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb"] Nov 01 16:15:28.092412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.078916 4911 scope.go:122] "RemoveContainer" containerID="ad149fe3fd8a99fdffa17ebd15d1613d77bcb86930eaaae06d8a64193c284139" Nov 01 16:15:28.092412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.087577 4911 generic.go:356] "Generic (PLEG): container finished" podID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerID="56adb93134d6f9d66cb9c5c4f664bbc9dba6c627f24365d72f8ac40fccc87bed" exitCode=60 Nov 01 16:15:28.092412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.087741 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerDied","Data":"56adb93134d6f9d66cb9c5c4f664bbc9dba6c627f24365d72f8ac40fccc87bed"} Nov 01 16:15:28.107540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.106395 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" event={"ID":"3eb9a3c2-532a-45f4-9489-b147b2969fab","Type":"ContainerStarted","Data":"cc6b4b3fdff81f563c15af6ca0fc93647e610818c9426d872a5ff1bf4c5292db"} Nov 01 16:15:28.107540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.106461 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" event={"ID":"3eb9a3c2-532a-45f4-9489-b147b2969fab","Type":"ContainerStarted","Data":"bfd6f03b98a1321284248cd03392cfa75b9a006b81454593383631fdab8babbf"} Nov 01 16:15:28.107540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.107315 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:28.130695 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.130666 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht"] Nov 01 16:15:28.163644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.163605 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f5dd93-29892495-dzbht"] Nov 01 16:15:28.188974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.186405 4911 scope.go:122] "RemoveContainer" containerID="acdcfa91b32500fe4c7e61a7963228c2016728cd191324dbcd783a348f7adb5d" Nov 01 16:15:28.206975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.206948 4911 scope.go:122] "RemoveContainer" containerID="a49e819a1307a3476a426a302cbd4b54c6dc341ba0a265bab2e5e981d4d52596" Nov 01 16:15:28.242061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.238304 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp"] Nov 01 16:15:28.272881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.272816 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" podStartSLOduration=17.272794906 podStartE2EDuration="17.272794906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:28.210342688 +0000 UTC m=+1668.354797330" watchObservedRunningTime="2026-11-01 16:15:28.272794906 +0000 UTC m=+1668.417249550" Nov 01 16:15:28.282231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.282194 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-784444-29892495-9kjrp"] Nov 01 16:15:28.391995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.391631 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:28.517798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.516343 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f085329-c40e-48bb-9978-fe2f8fb00695-kube-api-access-kfrld\" (UniqueName: \"kubernetes.io/projected/7f085329-c40e-48bb-9978-fe2f8fb00695-kube-api-access-kfrld\") pod \"7f085329-c40e-48bb-9978-fe2f8fb00695\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " Nov 01 16:15:28.518394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.516814 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f085329-c40e-48bb-9978-fe2f8fb00695-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085329-c40e-48bb-9978-fe2f8fb00695-maas-api-service-ca\") pod \"7f085329-c40e-48bb-9978-fe2f8fb00695\" (UID: \"7f085329-c40e-48bb-9978-fe2f8fb00695\") " Nov 01 16:15:28.519719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.519690 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f085329-c40e-48bb-9978-fe2f8fb00695-maas-api-service-ca" pod "7f085329-c40e-48bb-9978-fe2f8fb00695" (UID: "7f085329-c40e-48bb-9978-fe2f8fb00695"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:28.590665 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.590546 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f085329-c40e-48bb-9978-fe2f8fb00695-kube-api-access-kfrld" pod "7f085329-c40e-48bb-9978-fe2f8fb00695" (UID: "7f085329-c40e-48bb-9978-fe2f8fb00695"). InnerVolumeSpecName "kube-api-access-kfrld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:28.611338 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.611307 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:28.614124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.614098 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:28.618521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.618497 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:28.620862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.620837 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f085329-c40e-48bb-9978-fe2f8fb00695-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.620956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.620864 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfrld\" (UniqueName: \"kubernetes.io/projected/7f085329-c40e-48bb-9978-fe2f8fb00695-kube-api-access-kfrld\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.648431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.648392 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg"] Nov 01 16:15:28.649983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.649953 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:15:28.650185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650169 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:15:28.650279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650268 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.650351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650339 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.650433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650423 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:28.650546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650517 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:28.650649 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650637 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.650731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650721 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.650809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650800 4911 cpu_manager.go:414] "containerMap: removing container" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.650881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650871 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.650962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.650953 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:15:28.651030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651020 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:15:28.651145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651132 4911 cpu_manager.go:414] "containerMap: removing container" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.651220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651210 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.651298 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651289 4911 cpu_manager.go:414] "containerMap: removing container" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.651364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651355 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.651439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651430 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:15:28.651506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651497 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:15:28.651593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651583 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.651683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651673 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.651762 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651754 4911 cpu_manager.go:414] "containerMap: removing container" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.651829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651820 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.651914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651905 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:15:28.651984 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.651975 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:15:28.652117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652105 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:15:28.652212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652202 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:15:28.652292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652282 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.652362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652352 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.652437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652428 4911 cpu_manager.go:414] "containerMap: removing container" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.652507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652498 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.652581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652572 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.652653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652644 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.652727 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652719 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.652795 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652786 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.652870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652862 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:15:28.652937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.652928 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:15:28.653032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.653023 4911 cpu_manager.go:414] "containerMap: removing container" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.653149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.653137 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.654308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654287 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.654436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654424 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.654516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654506 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:15:28.654591 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654581 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.654666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654657 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:15:28.654740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654731 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7be1a72-b700-49aa-8b9c-d80cdc39168a" containerName="cleanup" Nov 01 16:15:28.654826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654816 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:15:28.654909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654899 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:15:28.654983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.654974 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:15:28.655076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655064 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.655150 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655142 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:15:28.655222 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655213 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.655304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655294 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.655384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655376 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:15:28.655460 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655451 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ba2926-33bf-4a15-b31d-f40ddd6c72c8" containerName="cleanup" Nov 01 16:15:28.655549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655540 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:15:28.655634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655626 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:15:28.655703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655694 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" containerName="cleanup" Nov 01 16:15:28.655774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.655766 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" containerName="cleanup" Nov 01 16:15:28.699141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.699105 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg"] Nov 01 16:15:28.699429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.699380 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:28.723633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.722795 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef06614d-8817-4af9-a659-2287b72ed0d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef06614d-8817-4af9-a659-2287b72ed0d0-maas-api-service-ca\") pod \"ef06614d-8817-4af9-a659-2287b72ed0d0\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " Nov 01 16:15:28.723633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.723085 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422af011-c613-4443-8047-d86cb44cd89f-kube-api-access-twn92\" (UniqueName: \"kubernetes.io/projected/422af011-c613-4443-8047-d86cb44cd89f-kube-api-access-twn92\") pod \"422af011-c613-4443-8047-d86cb44cd89f\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " Nov 01 16:15:28.723633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.723157 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed858f9a-835c-4c94-9866-5f28cdf2550b-kube-api-access-bcktf\" (UniqueName: \"kubernetes.io/projected/ed858f9a-835c-4c94-9866-5f28cdf2550b-kube-api-access-bcktf\") pod \"ed858f9a-835c-4c94-9866-5f28cdf2550b\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " Nov 01 16:15:28.723633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.723244 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef06614d-8817-4af9-a659-2287b72ed0d0-kube-api-access-nmjwn\" (UniqueName: \"kubernetes.io/projected/ef06614d-8817-4af9-a659-2287b72ed0d0-kube-api-access-nmjwn\") pod \"ef06614d-8817-4af9-a659-2287b72ed0d0\" (UID: \"ef06614d-8817-4af9-a659-2287b72ed0d0\") " Nov 01 16:15:28.723633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.723324 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422af011-c613-4443-8047-d86cb44cd89f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422af011-c613-4443-8047-d86cb44cd89f-maas-api-service-ca\") pod \"422af011-c613-4443-8047-d86cb44cd89f\" (UID: \"422af011-c613-4443-8047-d86cb44cd89f\") " Nov 01 16:15:28.723633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.723517 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed858f9a-835c-4c94-9866-5f28cdf2550b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed858f9a-835c-4c94-9866-5f28cdf2550b-maas-api-service-ca\") pod \"ed858f9a-835c-4c94-9866-5f28cdf2550b\" (UID: \"ed858f9a-835c-4c94-9866-5f28cdf2550b\") " Nov 01 16:15:28.724764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.724718 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed858f9a-835c-4c94-9866-5f28cdf2550b-maas-api-service-ca" pod "ed858f9a-835c-4c94-9866-5f28cdf2550b" (UID: "ed858f9a-835c-4c94-9866-5f28cdf2550b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:28.725167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.725144 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef06614d-8817-4af9-a659-2287b72ed0d0-maas-api-service-ca" pod "ef06614d-8817-4af9-a659-2287b72ed0d0" (UID: "ef06614d-8817-4af9-a659-2287b72ed0d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:28.725639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.725616 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422af011-c613-4443-8047-d86cb44cd89f-maas-api-service-ca" pod "422af011-c613-4443-8047-d86cb44cd89f" (UID: "422af011-c613-4443-8047-d86cb44cd89f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:28.732592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.732564 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef06614d-8817-4af9-a659-2287b72ed0d0-kube-api-access-nmjwn" pod "ef06614d-8817-4af9-a659-2287b72ed0d0" (UID: "ef06614d-8817-4af9-a659-2287b72ed0d0"). InnerVolumeSpecName "kube-api-access-nmjwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:28.733338 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.733304 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed858f9a-835c-4c94-9866-5f28cdf2550b-kube-api-access-bcktf" pod "ed858f9a-835c-4c94-9866-5f28cdf2550b" (UID: "ed858f9a-835c-4c94-9866-5f28cdf2550b"). InnerVolumeSpecName "kube-api-access-bcktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:28.735183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.735071 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422af011-c613-4443-8047-d86cb44cd89f-kube-api-access-twn92" pod "422af011-c613-4443-8047-d86cb44cd89f" (UID: "422af011-c613-4443-8047-d86cb44cd89f"). InnerVolumeSpecName "kube-api-access-twn92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:28.825705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.825668 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblj7\" (UniqueName: \"kubernetes.io/projected/71f7e203-4500-4fc7-a45b-166db555f0eb-kube-api-access-fblj7\") pod \"payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:28.825926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.825909 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7e203-4500-4fc7-a45b-166db555f0eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:28.826140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.826122 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422af011-c613-4443-8047-d86cb44cd89f-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.826179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.826143 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed858f9a-835c-4c94-9866-5f28cdf2550b-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.826179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.826154 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef06614d-8817-4af9-a659-2287b72ed0d0-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.826179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.826163 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twn92\" (UniqueName: \"kubernetes.io/projected/422af011-c613-4443-8047-d86cb44cd89f-kube-api-access-twn92\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.826179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.826173 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcktf\" (UniqueName: \"kubernetes.io/projected/ed858f9a-835c-4c94-9866-5f28cdf2550b-kube-api-access-bcktf\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.826304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.826184 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjwn\" (UniqueName: \"kubernetes.io/projected/ef06614d-8817-4af9-a659-2287b72ed0d0-kube-api-access-nmjwn\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:28.927853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.927810 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7e203-4500-4fc7-a45b-166db555f0eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:28.928146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.928125 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblj7\" (UniqueName: \"kubernetes.io/projected/71f7e203-4500-4fc7-a45b-166db555f0eb-kube-api-access-fblj7\") pod \"payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:28.928595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.928571 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7e203-4500-4fc7-a45b-166db555f0eb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:28.943009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:28.942925 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblj7\" (UniqueName: \"kubernetes.io/projected/71f7e203-4500-4fc7-a45b-166db555f0eb-kube-api-access-fblj7\") pod \"payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:29.043484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.043449 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:29.124066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.122863 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" Nov 01 16:15:29.124066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.122952 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs" event={"ID":"7f085329-c40e-48bb-9978-fe2f8fb00695","Type":"ContainerDied","Data":"e5d030ad9ac0b3ce1e158ecf789ae3c3142df64930c37d6acafecb4bdbe238ae"} Nov 01 16:15:29.124066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.123197 4911 scope.go:122] "RemoveContainer" containerID="4eb4eb2953a701a497796e3e4363ca475782fe61ff14d6839db0fabc22f46b7b" Nov 01 16:15:29.130096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.127507 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" event={"ID":"ed858f9a-835c-4c94-9866-5f28cdf2550b","Type":"ContainerDied","Data":"4777665d3476271bab27262f98a298176c6dc0f17306e0ee52d96de0900e880e"} Nov 01 16:15:29.130096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.127608 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll" Nov 01 16:15:29.140132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.136851 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" event={"ID":"422af011-c613-4443-8047-d86cb44cd89f","Type":"ContainerDied","Data":"0844dbab4ed29902390ddf0c9304718c1a9f4e9a734cdc8765dcdcb0742dba8e"} Nov 01 16:15:29.140132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.136956 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z" Nov 01 16:15:29.169308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.165536 4911 scope.go:122] "RemoveContainer" containerID="3b7d9328edf5041d95fe8824b1e5ce6b3c2ebe31ae6a0f02cf43d315360d37e9" Nov 01 16:15:29.169308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.165670 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" Nov 01 16:15:29.171853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.170982 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb" event={"ID":"ef06614d-8817-4af9-a659-2287b72ed0d0","Type":"ContainerDied","Data":"38bfd394499e86e05a046b449d60fc2c762479ede393d80807912ee8c2817522"} Nov 01 16:15:29.226012 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.225987 4911 scope.go:122] "RemoveContainer" containerID="f8bf14b76dd03afc0f182eaca287c0794126c17aa50c04dc1b8ce924d0fc91ea" Nov 01 16:15:29.240195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.240170 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs"] Nov 01 16:15:29.248023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.248003 4911 scope.go:122] "RemoveContainer" containerID="56adb93134d6f9d66cb9c5c4f664bbc9dba6c627f24365d72f8ac40fccc87bed" Nov 01 16:15:29.301712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.301676 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-10b00a-29892495-xh8cs"] Nov 01 16:15:29.326793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.323301 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb"] Nov 01 16:15:29.337397 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.337363 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f01de5-29892495-svcsb"] Nov 01 16:15:29.348493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.348458 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll"] Nov 01 16:15:29.359411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.359382 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2001b0-29892495-q5zll"] Nov 01 16:15:29.368182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.368103 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z"] Nov 01 16:15:29.375628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.375599 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892495-vc49z"] Nov 01 16:15:29.392643 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.384758 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg"] Nov 01 16:15:29.997370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.997330 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422af011-c613-4443-8047-d86cb44cd89f" path="/var/lib/kubelet/pods/422af011-c613-4443-8047-d86cb44cd89f/volumes" Nov 01 16:15:29.998138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.998113 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f085329-c40e-48bb-9978-fe2f8fb00695" path="/var/lib/kubelet/pods/7f085329-c40e-48bb-9978-fe2f8fb00695/volumes" Nov 01 16:15:29.998787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.998767 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eef4f46-6006-4038-8209-77ed34168412" path="/var/lib/kubelet/pods/9eef4f46-6006-4038-8209-77ed34168412/volumes" Nov 01 16:15:29.999973 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:29.999949 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5321715-5daf-4973-9218-0ce2ac3a0282" path="/var/lib/kubelet/pods/d5321715-5daf-4973-9218-0ce2ac3a0282/volumes" Nov 01 16:15:30.000613 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:30.000592 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" path="/var/lib/kubelet/pods/ed858f9a-835c-4c94-9866-5f28cdf2550b/volumes" Nov 01 16:15:30.006094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:30.001194 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" path="/var/lib/kubelet/pods/ef06614d-8817-4af9-a659-2287b72ed0d0/volumes" Nov 01 16:15:30.173228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:30.173176 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" event={"ID":"71f7e203-4500-4fc7-a45b-166db555f0eb","Type":"ContainerStarted","Data":"0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514"} Nov 01 16:15:30.173753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:30.173248 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" event={"ID":"71f7e203-4500-4fc7-a45b-166db555f0eb","Type":"ContainerStarted","Data":"5cc9df442657d0e738a952f6474423717f10a395bb2c25866fe31c1bc04f1958"} Nov 01 16:15:30.173753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:30.173456 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:15:30.193945 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:30.193886 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" podStartSLOduration=19.193868699 podStartE2EDuration="19.193868699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:30.191465547 +0000 UTC m=+1670.335920187" watchObservedRunningTime="2026-11-01 16:15:30.193868699 +0000 UTC m=+1670.338323342" Nov 01 16:15:31.959603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:31.959566 4911 scope.go:122] "RemoveContainer" containerID="42ebd1de6fa65a12e083c9954cbbc554c9fb027e349d8bf28f39d3817215d0ca" Nov 01 16:15:33.207523 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:33.207489 4911 generic.go:356] "Generic (PLEG): container finished" podID="fa496de7-fb8d-4329-9302-b6983e105838" containerID="4d9ff08b62d3416577cbe601bfb0c49f3e45eb0a3b84b5ea9abbfcae36fce3da" exitCode=60 Nov 01 16:15:33.207964 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:33.207564 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" event={"ID":"fa496de7-fb8d-4329-9302-b6983e105838","Type":"ContainerDied","Data":"4d9ff08b62d3416577cbe601bfb0c49f3e45eb0a3b84b5ea9abbfcae36fce3da"} Nov 01 16:15:33.207964 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:33.207629 4911 scope.go:122] "RemoveContainer" containerID="42ebd1de6fa65a12e083c9954cbbc554c9fb027e349d8bf28f39d3817215d0ca" Nov 01 16:15:33.208133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:33.208100 4911 scope.go:122] "RemoveContainer" containerID="4d9ff08b62d3416577cbe601bfb0c49f3e45eb0a3b84b5ea9abbfcae36fce3da" Nov 01 16:15:33.208549 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:33.208526 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6_odh-ai-gateway-infra(fa496de7-fb8d-4329-9302-b6983e105838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" podUID="fa496de7-fb8d-4329-9302-b6983e105838" Nov 01 16:15:33.959145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:33.959104 4911 scope.go:122] "RemoveContainer" containerID="01a705d7773e1887bc01f4541f74128ccd6d79a73dc36830467abb20ba234f1c" Nov 01 16:15:34.328064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.328006 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6"] Nov 01 16:15:34.586584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.586553 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:34.774035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.774004 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa496de7-fb8d-4329-9302-b6983e105838-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa496de7-fb8d-4329-9302-b6983e105838-maas-api-service-ca\") pod \"fa496de7-fb8d-4329-9302-b6983e105838\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " Nov 01 16:15:34.774275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.774257 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa496de7-fb8d-4329-9302-b6983e105838-kube-api-access-sv52g\" (UniqueName: \"kubernetes.io/projected/fa496de7-fb8d-4329-9302-b6983e105838-kube-api-access-sv52g\") pod \"fa496de7-fb8d-4329-9302-b6983e105838\" (UID: \"fa496de7-fb8d-4329-9302-b6983e105838\") " Nov 01 16:15:34.774411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.774387 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa496de7-fb8d-4329-9302-b6983e105838-maas-api-service-ca" pod "fa496de7-fb8d-4329-9302-b6983e105838" (UID: "fa496de7-fb8d-4329-9302-b6983e105838"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:34.774979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.774962 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa496de7-fb8d-4329-9302-b6983e105838-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:34.776480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.776459 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa496de7-fb8d-4329-9302-b6983e105838-kube-api-access-sv52g" pod "fa496de7-fb8d-4329-9302-b6983e105838" (UID: "fa496de7-fb8d-4329-9302-b6983e105838"). InnerVolumeSpecName "kube-api-access-sv52g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:34.876244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:34.876208 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv52g\" (UniqueName: \"kubernetes.io/projected/fa496de7-fb8d-4329-9302-b6983e105838-kube-api-access-sv52g\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:35.272626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.272593 4911 generic.go:356] "Generic (PLEG): container finished" podID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerID="0173aa5a1c30569bb8a66c9293eaa3f205211bbb00b6099b566a767deadf6a40" exitCode=60 Nov 01 16:15:35.272790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.272663 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" event={"ID":"b292b099-0e0e-4f27-8fb7-db1a2e22e768","Type":"ContainerDied","Data":"0173aa5a1c30569bb8a66c9293eaa3f205211bbb00b6099b566a767deadf6a40"} Nov 01 16:15:35.272790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.272726 4911 scope.go:122] "RemoveContainer" containerID="01a705d7773e1887bc01f4541f74128ccd6d79a73dc36830467abb20ba234f1c" Nov 01 16:15:35.273317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.273291 4911 scope.go:122] "RemoveContainer" containerID="0173aa5a1c30569bb8a66c9293eaa3f205211bbb00b6099b566a767deadf6a40" Nov 01 16:15:35.273819 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:35.273793 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft_odh-ai-gateway-infra(b292b099-0e0e-4f27-8fb7-db1a2e22e768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" Nov 01 16:15:35.274411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.274391 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" Nov 01 16:15:35.274506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.274430 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6" event={"ID":"fa496de7-fb8d-4329-9302-b6983e105838","Type":"ContainerDied","Data":"7ca4366b4f0438928f20f36039d2d58c8406608d147c4d4a962b94f174658acb"} Nov 01 16:15:35.292687 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.292664 4911 scope.go:122] "RemoveContainer" containerID="4d9ff08b62d3416577cbe601bfb0c49f3e45eb0a3b84b5ea9abbfcae36fce3da" Nov 01 16:15:35.310779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.310722 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6"] Nov 01 16:15:35.316668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.316642 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-04f248-29892495-xsdf6"] Nov 01 16:15:35.971149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:35.971111 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa496de7-fb8d-4329-9302-b6983e105838" path="/var/lib/kubelet/pods/fa496de7-fb8d-4329-9302-b6983e105838/volumes" Nov 01 16:15:36.315619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.315575 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft"] Nov 01 16:15:36.509536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.509504 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:36.633864 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.632821 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b292b099-0e0e-4f27-8fb7-db1a2e22e768-kube-api-access-xscnk\" (UniqueName: \"kubernetes.io/projected/b292b099-0e0e-4f27-8fb7-db1a2e22e768-kube-api-access-xscnk\") pod \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " Nov 01 16:15:36.633864 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.633456 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b292b099-0e0e-4f27-8fb7-db1a2e22e768-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b292b099-0e0e-4f27-8fb7-db1a2e22e768-maas-api-service-ca\") pod \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\" (UID: \"b292b099-0e0e-4f27-8fb7-db1a2e22e768\") " Nov 01 16:15:36.635028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.634996 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b292b099-0e0e-4f27-8fb7-db1a2e22e768-maas-api-service-ca" pod "b292b099-0e0e-4f27-8fb7-db1a2e22e768" (UID: "b292b099-0e0e-4f27-8fb7-db1a2e22e768"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:36.645759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.641679 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b292b099-0e0e-4f27-8fb7-db1a2e22e768-kube-api-access-xscnk" pod "b292b099-0e0e-4f27-8fb7-db1a2e22e768" (UID: "b292b099-0e0e-4f27-8fb7-db1a2e22e768"). InnerVolumeSpecName "kube-api-access-xscnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:36.736146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.736106 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xscnk\" (UniqueName: \"kubernetes.io/projected/b292b099-0e0e-4f27-8fb7-db1a2e22e768-kube-api-access-xscnk\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:36.736146 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:36.736137 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b292b099-0e0e-4f27-8fb7-db1a2e22e768-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:37.300902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:37.300869 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" Nov 01 16:15:37.301414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:37.300900 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft" event={"ID":"b292b099-0e0e-4f27-8fb7-db1a2e22e768","Type":"ContainerDied","Data":"7ef2d15de5311bffc88e32dfc1f92e459d0d0c4529022f12fdf9b0449072cf2b"} Nov 01 16:15:37.301414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:37.300977 4911 scope.go:122] "RemoveContainer" containerID="0173aa5a1c30569bb8a66c9293eaa3f205211bbb00b6099b566a767deadf6a40" Nov 01 16:15:37.335814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:37.335776 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft"] Nov 01 16:15:37.340832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:37.340805 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f47890-29892495-284ft"] Nov 01 16:15:37.975242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:37.975191 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" path="/var/lib/kubelet/pods/b292b099-0e0e-4f27-8fb7-db1a2e22e768/volumes" Nov 01 16:15:40.181129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:40.181097 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:15:40.869809 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:15:40.869768 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:15:41.196096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:15:41.195957 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:16:01.819174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.819140 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd"] Nov 01 16:16:01.819986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.819969 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.819987 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820005 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820010 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820020 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820025 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820061 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820069 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820086 4911 cpu_manager.go:414] "containerMap: removing container" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820093 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820113 4911 cpu_manager.go:414] "containerMap: removing container" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820119 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820130 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820135 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eef4f46-6006-4038-8209-77ed34168412" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820150 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820156 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820164 4911 cpu_manager.go:414] "containerMap: removing container" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820169 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820334 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820356 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820366 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="0324204c-7d4f-4644-aa70-39e7e8a0e7e7" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820380 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:01.820395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820397 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed858f9a-835c-4c94-9866-5f28cdf2550b" containerName="cleanup" Nov 01 16:16:01.820793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820409 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="422af011-c613-4443-8047-d86cb44cd89f" containerName="cleanup" Nov 01 16:16:01.820793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820423 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef06614d-8817-4af9-a659-2287b72ed0d0" containerName="cleanup" Nov 01 16:16:01.820793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.820443 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:01.826643 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.825405 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:01.835739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.835710 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a4240a\"/\"openshift-service-ca.crt\"" Nov 01 16:16:01.835875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.835840 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a4240a\"/\"kube-root-ca.crt\"" Nov 01 16:16:01.842386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.842357 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd"] Nov 01 16:16:01.857466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.857434 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a4240a\"/\"test-model-a4240a-kserve-self-signed-certs\"" Nov 01 16:16:01.857631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.857531 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a4240a\"/\"default-dockercfg-k2lth\"" Nov 01 16:16:01.944644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.944602 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-dshm\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:01.944823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.944652 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-tmp-dir\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:01.944823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.944801 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2cg\" (UniqueName: \"kubernetes.io/projected/97b70d41-ad60-43ee-8396-b2c7610555d9-kube-api-access-wx2cg\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:01.944981 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.944960 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-home\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:01.945094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.945007 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b70d41-ad60-43ee-8396-b2c7610555d9-tls-certs\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:01.945162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:01.945139 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-model-cache\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.049068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.047397 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-dshm\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.049068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.047482 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-tmp-dir\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.049068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.047516 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2cg\" (UniqueName: \"kubernetes.io/projected/97b70d41-ad60-43ee-8396-b2c7610555d9-kube-api-access-wx2cg\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.049068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.047689 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-home\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.049068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.047742 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b70d41-ad60-43ee-8396-b2c7610555d9-tls-certs\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.049068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.047791 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-model-cache\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.049068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.048444 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-model-cache\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.072107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.065745 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-tmp-dir\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.072107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.066017 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-home\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.088059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.087285 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-dshm\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.096064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.093905 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2cg\" (UniqueName: \"kubernetes.io/projected/97b70d41-ad60-43ee-8396-b2c7610555d9-kube-api-access-wx2cg\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.138957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.138909 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b70d41-ad60-43ee-8396-b2c7610555d9-tls-certs\") pod \"test-model-a4240a-kserve-5585b759ff-657vd\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.163821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.163079 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:02.549138 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:16:02.549095 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97b70d41_ad60_43ee_8396_b2c7610555d9.slice/crio-27149370bb47bf647ad71b98ae7352a7a383561dfa2da96518d5fe96f593073a WatchSource:0}: Error finding container 27149370bb47bf647ad71b98ae7352a7a383561dfa2da96518d5fe96f593073a: Status 404 returned error can't find the container with id 27149370bb47bf647ad71b98ae7352a7a383561dfa2da96518d5fe96f593073a Nov 01 16:16:02.550646 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:02.550619 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd"] Nov 01 16:16:03.565869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:03.565828 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" event={"ID":"97b70d41-ad60-43ee-8396-b2c7610555d9","Type":"ContainerStarted","Data":"dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da"} Nov 01 16:16:03.566312 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:03.565884 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" event={"ID":"97b70d41-ad60-43ee-8396-b2c7610555d9","Type":"ContainerStarted","Data":"27149370bb47bf647ad71b98ae7352a7a383561dfa2da96518d5fe96f593073a"} Nov 01 16:16:03.586009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:03.585951 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" podStartSLOduration=2.585933745 podStartE2EDuration="2.585933745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:16:03.580782072 +0000 UTC m=+1703.725236717" watchObservedRunningTime="2026-11-01 16:16:03.585933745 +0000 UTC m=+1703.730388389" Nov 01 16:16:04.163636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:04.163594 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:10.877746 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:16:10.877706 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:16:12.164519 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:12.164483 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:12.178117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:12.178078 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:12.670405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:12.670374 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:16:16.183018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.182983 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4"] Nov 01 16:16:16.184288 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.184263 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b292b099-0e0e-4f27-8fb7-db1a2e22e768" containerName="cleanup" Nov 01 16:16:16.184564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.184511 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa496de7-fb8d-4329-9302-b6983e105838" containerName="cleanup" Nov 01 16:16:16.190206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.190180 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.200393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.200362 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0c091d\"/\"test-model-0c091d-kserve-self-signed-certs\"" Nov 01 16:16:16.200598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.200406 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0c091d\"/\"openshift-service-ca.crt\"" Nov 01 16:16:16.200598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.200506 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0c091d\"/\"default-dockercfg-hgfp8\"" Nov 01 16:16:16.201503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.201477 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0c091d\"/\"kube-root-ca.crt\"" Nov 01 16:16:16.228018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.227983 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4"] Nov 01 16:16:16.348406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.348361 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-model-cache\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.348597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.348561 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2908c72c-772e-4003-a90b-7ed121e92979-tls-certs\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.348759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.348740 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-dshm\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.348823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.348810 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2pl\" (UniqueName: \"kubernetes.io/projected/2908c72c-772e-4003-a90b-7ed121e92979-kube-api-access-jl2pl\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.348954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.348929 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-home\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.349102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.349078 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-tmp-dir\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.450959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.450861 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-tmp-dir\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.451179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.450971 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-model-cache\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.451179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.451132 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2908c72c-772e-4003-a90b-7ed121e92979-tls-certs\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.451314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.451294 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-dshm\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.451413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.451385 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-tmp-dir\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.451483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.451446 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-model-cache\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.452419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.452388 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2pl\" (UniqueName: \"kubernetes.io/projected/2908c72c-772e-4003-a90b-7ed121e92979-kube-api-access-jl2pl\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.452564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.452528 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-home\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.452958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.452932 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-home\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.454292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.454265 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2908c72c-772e-4003-a90b-7ed121e92979-tls-certs\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.454833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.454806 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-dshm\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.492251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.490688 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2pl\" (UniqueName: \"kubernetes.io/projected/2908c72c-772e-4003-a90b-7ed121e92979-kube-api-access-jl2pl\") pod \"test-model-0c091d-kserve-5c547c9997-cq9n4\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:16.523692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:16.523321 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:17.025405 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:16:17.025371 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2908c72c_772e_4003_a90b_7ed121e92979.slice/crio-82f7c662babd204a55ea5b46988cfb920114ae22251dd91caa3272af49336bd4 WatchSource:0}: Error finding container 82f7c662babd204a55ea5b46988cfb920114ae22251dd91caa3272af49336bd4: Status 404 returned error can't find the container with id 82f7c662babd204a55ea5b46988cfb920114ae22251dd91caa3272af49336bd4 Nov 01 16:16:17.025758 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:17.025738 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4"] Nov 01 16:16:17.780909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:17.780870 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" event={"ID":"2908c72c-772e-4003-a90b-7ed121e92979","Type":"ContainerStarted","Data":"980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39"} Nov 01 16:16:17.781463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:17.780919 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" event={"ID":"2908c72c-772e-4003-a90b-7ed121e92979","Type":"ContainerStarted","Data":"82f7c662babd204a55ea5b46988cfb920114ae22251dd91caa3272af49336bd4"} Nov 01 16:16:17.802742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:17.802691 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" podStartSLOduration=1.802677691 podStartE2EDuration="1.802677691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:16:17.796828314 +0000 UTC m=+1717.941282955" watchObservedRunningTime="2026-11-01 16:16:17.802677691 +0000 UTC m=+1717.947132332" Nov 01 16:16:18.523509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:18.523471 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:26.523861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:26.523822 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:26.536996 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:26.536968 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:26.878774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:26.878692 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:16:40.875176 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:16:40.875132 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:16:44.159154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.159119 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s"] Nov 01 16:16:44.164465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.164441 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.170017 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.169982 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ae79df\"" Nov 01 16:16:44.171408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.171364 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ae79df-dockercfg-tkqc9\"" Nov 01 16:16:44.171542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.171473 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s"] Nov 01 16:16:44.217542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.217506 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.217850 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.217789 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwm9\" (UniqueName: \"kubernetes.io/projected/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-kube-api-access-2jwm9\") pod \"payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.324483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.324413 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwm9\" (UniqueName: \"kubernetes.io/projected/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-kube-api-access-2jwm9\") pod \"payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.324954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.324931 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.387065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.375478 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.461728 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.461644 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwm9\" (UniqueName: \"kubernetes.io/projected/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-kube-api-access-2jwm9\") pod \"payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.493154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.493123 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:44.860993 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:16:44.860964 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff14d30a_77f0_4394_bfd5_bcc60bdd866f.slice/crio-f29d92c1ae6411d2f7b00f1c8df6fab1b4d4c927e442b21798445455d3f98ed8 WatchSource:0}: Error finding container f29d92c1ae6411d2f7b00f1c8df6fab1b4d4c927e442b21798445455d3f98ed8: Status 404 returned error can't find the container with id f29d92c1ae6411d2f7b00f1c8df6fab1b4d4c927e442b21798445455d3f98ed8 Nov 01 16:16:44.862730 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:44.862698 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s"] Nov 01 16:16:45.131432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:45.131337 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" event={"ID":"ff14d30a-77f0-4394-bfd5-bcc60bdd866f","Type":"ContainerStarted","Data":"6d7ad9ad43324a1c0c5485006196a55631fbd5f0143de1ff46205017bb6dddd9"} Nov 01 16:16:45.131432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:45.131397 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" event={"ID":"ff14d30a-77f0-4394-bfd5-bcc60bdd866f","Type":"ContainerStarted","Data":"f29d92c1ae6411d2f7b00f1c8df6fab1b4d4c927e442b21798445455d3f98ed8"} Nov 01 16:16:45.131432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:45.131436 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:16:45.156769 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:45.156712 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" podStartSLOduration=1.1566945 podStartE2EDuration="1.1566945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:16:45.150689475 +0000 UTC m=+1745.295144117" watchObservedRunningTime="2026-11-01 16:16:45.1566945 +0000 UTC m=+1745.301149142" Nov 01 16:16:56.143373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:16:56.143294 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:17:10.871769 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:17:10.871708 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:17:40.514035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:17:40.514005 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:17:40.532013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:17:40.531984 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:17:40.869100 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:17:40.868995 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:18:00.772318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:00.772275 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z"] Nov 01 16:18:00.772830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:00.772505 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" containerID="cri-o://710013b9872a8d2bd0c5bc7b16b179feb9f053d3dd4cb7d81cc8ea3ba54e6793" gracePeriod=30 Nov 01 16:18:06.159353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.159266 4911 generic.go:356] "Generic (PLEG): container finished" podID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerID="710013b9872a8d2bd0c5bc7b16b179feb9f053d3dd4cb7d81cc8ea3ba54e6793" exitCode=0 Nov 01 16:18:06.159353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.159327 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" event={"ID":"542c63ce-b126-4f1d-8eaf-d5ca3e087df6","Type":"ContainerDied","Data":"710013b9872a8d2bd0c5bc7b16b179feb9f053d3dd4cb7d81cc8ea3ba54e6793"} Nov 01 16:18:06.537581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.537552 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:18:06.547963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.547890 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-socket\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.547963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.547973 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-envoy\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548001 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-podinfo\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548024 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-certs\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548203 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-socket" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:06.548242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548210 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istiod-ca-cert\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548383 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-token\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-token\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548421 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-credential-socket\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548453 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-certs" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:06.548533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548462 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-kube-api-access-lkf9l\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-kube-api-access-lkf9l\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548547 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istiod-ca-cert" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:06.548804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548766 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-data\") pod \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\" (UID: \"542c63ce-b126-4f1d-8eaf-d5ca3e087df6\") " Nov 01 16:18:06.548919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.548822 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-credential-socket" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:06.549215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.549188 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-data" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:06.549743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.549718 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.549848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.549749 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.549848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.549765 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.549848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.549780 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.549848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.549795 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.550723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.550698 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-envoy" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:06.550819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.550751 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-token" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:06.550951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.550929 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-podinfo" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:06.550996 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.550937 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-kube-api-access-lkf9l" pod "542c63ce-b126-4f1d-8eaf-d5ca3e087df6" (UID: "542c63ce-b126-4f1d-8eaf-d5ca3e087df6"). InnerVolumeSpecName "kube-api-access-lkf9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:06.652113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.652065 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.652113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.652109 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.652113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.652123 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:06.652358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:06.652137 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkf9l\" (UniqueName: \"kubernetes.io/projected/542c63ce-b126-4f1d-8eaf-d5ca3e087df6-kube-api-access-lkf9l\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:07.171586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:07.171550 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" Nov 01 16:18:07.172072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:07.171580 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z" event={"ID":"542c63ce-b126-4f1d-8eaf-d5ca3e087df6","Type":"ContainerDied","Data":"1599bfb6f8d2af965a35d823a6d1950dded9e25eb71a23267c2bd204af2bd1e9"} Nov 01 16:18:07.172217 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:07.172118 4911 scope.go:122] "RemoveContainer" containerID="710013b9872a8d2bd0c5bc7b16b179feb9f053d3dd4cb7d81cc8ea3ba54e6793" Nov 01 16:18:07.204661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:07.204612 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z"] Nov 01 16:18:07.209952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:07.209925 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-fp42z"] Nov 01 16:18:07.970598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:07.970564 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" path="/var/lib/kubelet/pods/542c63ce-b126-4f1d-8eaf-d5ca3e087df6/volumes" Nov 01 16:18:10.927090 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:10.926964 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:18:15.627026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:15.626983 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r"] Nov 01 16:18:15.627599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:15.627263 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" containerID="cri-o://3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e" gracePeriod=30 Nov 01 16:18:15.634754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:15.634717 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq"] Nov 01 16:18:15.635032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:15.634984 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" containerID="cri-o://1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91" gracePeriod=30 Nov 01 16:18:15.973653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:15.973559 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s"] Nov 01 16:18:15.973830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:15.973786 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" containerID="cri-o://2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c" gracePeriod=30 Nov 01 16:18:18.706607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:18.706569 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh"] Nov 01 16:18:21.009056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.004394 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:18:21.091537 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.091498 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:18:21.107814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.107780 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-kube-api-access-kl9dc\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-kube-api-access-kl9dc\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.107968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.107947 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-envoy\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108033 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-certs\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108122 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-data\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108274 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-socket\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108316 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istiod-ca-cert\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108436 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-token\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-token\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108468 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-podinfo\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108514 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-credential-socket\") pod \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\" (UID: \"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3\") " Nov 01 16:18:21.108907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.108880 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-socket" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.109140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109112 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-data" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.109140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109129 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-certs" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.109273 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109143 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-credential-socket" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.109632 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109604 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istiod-ca-cert" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:21.109892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109809 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.109892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109833 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.109892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109844 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.109892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109856 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.109892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.109868 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.135414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.133371 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-podinfo" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:21.135414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.133583 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-token" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.135414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.133678 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-envoy" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.162876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.162835 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-kube-api-access-kl9dc" pod "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" (UID: "a07ef0c7-b67d-4849-a0f1-b4b84aa022e3"). InnerVolumeSpecName "kube-api-access-kl9dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.210661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.210624 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-podinfo\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.210822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.210704 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-credential-socket\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.210822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.210756 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-certs\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.210822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.210785 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-socket\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.211318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.210866 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-token\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-token\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.211318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.210990 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60af24b8-930b-48af-a4d9-0ee884f97bb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af24b8-930b-48af-a4d9-0ee884f97bb3-istiod-ca-cert\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.211318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211068 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-kube-api-access-dml7r\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-kube-api-access-dml7r\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.211318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211071 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-credential-socket" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.211318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211118 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-data\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.211318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211117 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-socket" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.211318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211254 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-envoy\") pod \"60af24b8-930b-48af-a4d9-0ee884f97bb3\" (UID: \"60af24b8-930b-48af-a4d9-0ee884f97bb3\") " Nov 01 16:18:21.211688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211414 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60af24b8-930b-48af-a4d9-0ee884f97bb3-istiod-ca-cert" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:21.211688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211483 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-certs" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.211842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.211816 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-data" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.212279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212251 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60af24b8-930b-48af-a4d9-0ee884f97bb3-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212284 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212300 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212315 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212331 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl9dc\" (UniqueName: \"kubernetes.io/projected/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-kube-api-access-kl9dc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212346 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212359 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212374 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.212388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.212388 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.214469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.214442 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-token" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.214574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.214532 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-envoy" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.215405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.215378 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-kube-api-access-dml7r" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "kube-api-access-dml7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.218086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.218056 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-podinfo" pod "60af24b8-930b-48af-a4d9-0ee884f97bb3" (UID: "60af24b8-930b-48af-a4d9-0ee884f97bb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:21.314705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.314668 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.314705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.314699 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dml7r\" (UniqueName: \"kubernetes.io/projected/60af24b8-930b-48af-a4d9-0ee884f97bb3-kube-api-access-dml7r\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.314705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.314711 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.314985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.314720 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60af24b8-930b-48af-a4d9-0ee884f97bb3-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.326328 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.326299 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:18:21.335660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.335626 4911 generic.go:356] "Generic (PLEG): container finished" podID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerID="1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91" exitCode=0 Nov 01 16:18:21.335810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.335695 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" Nov 01 16:18:21.335810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.335712 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" event={"ID":"60af24b8-930b-48af-a4d9-0ee884f97bb3","Type":"ContainerDied","Data":"1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91"} Nov 01 16:18:21.335919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.335815 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq" event={"ID":"60af24b8-930b-48af-a4d9-0ee884f97bb3","Type":"ContainerDied","Data":"98f49047cfda77da8aecce8b136f7a5d16f2c5f2dda9ecd969ca8d595d26f2c6"} Nov 01 16:18:21.335919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.335844 4911 scope.go:122] "RemoveContainer" containerID="1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91" Nov 01 16:18:21.338329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.337500 4911 generic.go:356] "Generic (PLEG): container finished" podID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerID="2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c" exitCode=0 Nov 01 16:18:21.338329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.337556 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" Nov 01 16:18:21.338329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.337686 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" event={"ID":"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df","Type":"ContainerDied","Data":"2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c"} Nov 01 16:18:21.338329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.337742 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s" event={"ID":"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df","Type":"ContainerDied","Data":"dd4b952b6c8e04d4a7bfebdd88f9626a28eb9b95d5a9dc35453944b47db10676"} Nov 01 16:18:21.339498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.339473 4911 generic.go:356] "Generic (PLEG): container finished" podID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerID="3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e" exitCode=0 Nov 01 16:18:21.339639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.339546 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" Nov 01 16:18:21.339639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.339559 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" event={"ID":"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3","Type":"ContainerDied","Data":"3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e"} Nov 01 16:18:21.339639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.339607 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r" event={"ID":"a07ef0c7-b67d-4849-a0f1-b4b84aa022e3","Type":"ContainerDied","Data":"fe8c25258ace5cf5e3c201a7f64ad614b5e099e5de621cdffd4699eb426197a6"} Nov 01 16:18:21.354199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.354176 4911 scope.go:122] "RemoveContainer" containerID="1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91" Nov 01 16:18:21.354801 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:21.354678 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91\": container with ID starting with 1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91 not found: ID does not exist" containerID="1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91" Nov 01 16:18:21.354801 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.354713 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91"} err="failed to get container status \"1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91\": rpc error: code = NotFound desc = could not find container \"1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91\": container with ID starting with 1c726566c940f12d99d48210abf8ef6c3fde6eb6009b18c97817f8525b1b6e91 not found: ID does not exist" Nov 01 16:18:21.354801 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.354736 4911 scope.go:122] "RemoveContainer" containerID="2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c" Nov 01 16:18:21.382602 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.382574 4911 scope.go:122] "RemoveContainer" containerID="2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c" Nov 01 16:18:21.383296 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:21.383260 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c\": container with ID starting with 2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c not found: ID does not exist" containerID="2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c" Nov 01 16:18:21.383433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.383307 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c"} err="failed to get container status \"2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c\": rpc error: code = NotFound desc = could not find container \"2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c\": container with ID starting with 2019de1ce3bae9c382f6c2d4ea0f89fbbe6c5c2c901ac582cd5e4325bedcdd4c not found: ID does not exist" Nov 01 16:18:21.383433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.383330 4911 scope.go:122] "RemoveContainer" containerID="3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e" Nov 01 16:18:21.408027 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.407999 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r"] Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415386 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-podinfo\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415525 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-envoy\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415568 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istiod-ca-cert\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415599 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-data\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415681 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-socket\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415873 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-kube-api-access-7k52f\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-kube-api-access-7k52f\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415912 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-credential-socket\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415944 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-certs\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.417716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.415973 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-token\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-token\") pod \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\" (UID: \"662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df\") " Nov 01 16:18:21.419621 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.419374 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-data" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.420375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.420353 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-credential-socket" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.420652 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.420637 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-certs" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.421003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.420987 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istiod-ca-cert" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:21.421317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.421274 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-socket" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.423755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.423684 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-64w4r"] Nov 01 16:18:21.430087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.429230 4911 scope.go:122] "RemoveContainer" containerID="3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e" Nov 01 16:18:21.431705 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:21.430330 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e\": container with ID starting with 3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e not found: ID does not exist" containerID="3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e" Nov 01 16:18:21.431705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.430366 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e"} err="failed to get container status \"3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e\": rpc error: code = NotFound desc = could not find container \"3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e\": container with ID starting with 3f9a8cd5f822fdac1e9938a3b52b456f966016e0e3e9f7d3d68d1b3964bddd7e not found: ID does not exist" Nov 01 16:18:21.435572 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.435510 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq"] Nov 01 16:18:21.436137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.436109 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-podinfo" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:21.436411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.436373 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-token" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.444678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.444600 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-7vzcq"] Nov 01 16:18:21.445681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.445639 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-kube-api-access-7k52f" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "kube-api-access-7k52f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.451494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.451447 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-envoy" pod "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" (UID: "662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.518148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518113 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k52f\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-kube-api-access-7k52f\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518146 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518156 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518167 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518177 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518187 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518196 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518204 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.518385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.518212 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.586108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.586069 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6"] Nov 01 16:18:21.586358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.586331 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" containerID="cri-o://6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d" gracePeriod=30 Nov 01 16:18:21.723237 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.723149 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s"] Nov 01 16:18:21.730586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.730547 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-5xx6s"] Nov 01 16:18:21.975344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.975248 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" path="/var/lib/kubelet/pods/60af24b8-930b-48af-a4d9-0ee884f97bb3/volumes" Nov 01 16:18:21.976092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.976066 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" path="/var/lib/kubelet/pods/662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df/volumes" Nov 01 16:18:21.976734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:21.976712 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" path="/var/lib/kubelet/pods/a07ef0c7-b67d-4849-a0f1-b4b84aa022e3/volumes" Nov 01 16:18:26.916484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:26.916453 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:18:27.042065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.026437 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-podinfo\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195182 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-socket\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195392 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-envoy\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195485 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istiod-ca-cert\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195515 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-credential-socket\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195560 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-certs\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195852 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-data\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195895 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-token\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-token\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.195922 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-kube-api-access-79q6z\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-kube-api-access-79q6z\") pod \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\" (UID: \"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1\") " Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.125558 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-podinfo" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.199485 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istiod-ca-cert" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:27.201114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.200999 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-data" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:27.201853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.201219 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-credential-socket" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:27.201853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.201621 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-certs" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:27.251003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.223961 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-socket" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:27.251003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.238239 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-kube-api-access-79q6z" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "kube-api-access-79q6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:27.251003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.238337 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-envoy" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:27.251003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.248485 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-token" pod "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" (UID: "f16c4f4e-8946-4ec7-a5d2-2d08c925dad1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:27.299802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299695 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.299802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299736 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.299802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299751 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.299802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299764 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.299802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299778 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.299802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299790 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.299802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299805 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79q6z\" (UniqueName: \"kubernetes.io/projected/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-kube-api-access-79q6z\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.300238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299818 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.300238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.299832 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:27.409760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.409715 4911 generic.go:356] "Generic (PLEG): container finished" podID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerID="6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d" exitCode=0 Nov 01 16:18:27.409760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.409764 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" Nov 01 16:18:27.410085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.409833 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" event={"ID":"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1","Type":"ContainerDied","Data":"6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d"} Nov 01 16:18:27.410085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.409883 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6" event={"ID":"f16c4f4e-8946-4ec7-a5d2-2d08c925dad1","Type":"ContainerDied","Data":"fded38dcbc8eb84ed307373291f19b2f72ca76529c3e2c35cbc04624dad34d5b"} Nov 01 16:18:27.410085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.409912 4911 scope.go:122] "RemoveContainer" containerID="6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d" Nov 01 16:18:27.425397 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.425372 4911 scope.go:122] "RemoveContainer" containerID="6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d" Nov 01 16:18:27.425764 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:27.425742 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d\": container with ID starting with 6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d not found: ID does not exist" containerID="6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d" Nov 01 16:18:27.425824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.425773 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d"} err="failed to get container status \"6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d\": rpc error: code = NotFound desc = could not find container \"6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d\": container with ID starting with 6aafe6712c3de5870bfb65757ee717de1b6dde6e4415f02db05ea1eb309ea26d not found: ID does not exist" Nov 01 16:18:27.445396 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.445355 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6"] Nov 01 16:18:27.454187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.454102 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-76nz6"] Nov 01 16:18:27.972864 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:27.972828 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" path="/var/lib/kubelet/pods/f16c4f4e-8946-4ec7-a5d2-2d08c925dad1/volumes" Nov 01 16:18:29.049920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:29.049877 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc"] Nov 01 16:18:29.065867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:29.065831 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq"] Nov 01 16:18:33.750270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:33.750224 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" podUID="c34c58b2-eb0e-4628-ba46-98207d6443ee" containerName="main" containerID="cri-o://1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570" gracePeriod=45 Nov 01 16:18:34.050457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.050403 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" podUID="3097fcd8-308a-4974-86f8-28bf25d8ee17" containerName="payload-processing" containerID="cri-o://7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7" gracePeriod=25 Nov 01 16:18:34.067098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.067009 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" podUID="0f02b673-119f-412d-85db-d0bd27362550" containerName="payload-pre-processing" containerID="cri-o://8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f" gracePeriod=25 Nov 01 16:18:34.090972 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.090944 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:18:34.279155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279119 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-model-cache\") pod \"c34c58b2-eb0e-4628-ba46-98207d6443ee\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " Nov 01 16:18:34.279352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279243 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-dshm\") pod \"c34c58b2-eb0e-4628-ba46-98207d6443ee\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " Nov 01 16:18:34.279352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279307 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34c58b2-eb0e-4628-ba46-98207d6443ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/c34c58b2-eb0e-4628-ba46-98207d6443ee-tls-certs\") pod \"c34c58b2-eb0e-4628-ba46-98207d6443ee\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " Nov 01 16:18:34.279352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279334 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-home\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-home\") pod \"c34c58b2-eb0e-4628-ba46-98207d6443ee\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " Nov 01 16:18:34.279553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279445 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-model-cache" pod "c34c58b2-eb0e-4628-ba46-98207d6443ee" (UID: "c34c58b2-eb0e-4628-ba46-98207d6443ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:34.279553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279541 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-tmp-dir\") pod \"c34c58b2-eb0e-4628-ba46-98207d6443ee\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " Nov 01 16:18:34.279677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279654 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-home" pod "c34c58b2-eb0e-4628-ba46-98207d6443ee" (UID: "c34c58b2-eb0e-4628-ba46-98207d6443ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:34.279677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279667 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34c58b2-eb0e-4628-ba46-98207d6443ee-kube-api-access-ht7ws\" (UniqueName: \"kubernetes.io/projected/c34c58b2-eb0e-4628-ba46-98207d6443ee-kube-api-access-ht7ws\") pod \"c34c58b2-eb0e-4628-ba46-98207d6443ee\" (UID: \"c34c58b2-eb0e-4628-ba46-98207d6443ee\") " Nov 01 16:18:34.279895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.279873 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-tmp-dir" pod "c34c58b2-eb0e-4628-ba46-98207d6443ee" (UID: "c34c58b2-eb0e-4628-ba46-98207d6443ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:34.280483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.280459 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:34.280554 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.280490 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:34.280554 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.280506 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:34.281684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.281659 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-dshm" pod "c34c58b2-eb0e-4628-ba46-98207d6443ee" (UID: "c34c58b2-eb0e-4628-ba46-98207d6443ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:34.281805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.281760 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34c58b2-eb0e-4628-ba46-98207d6443ee-tls-certs" pod "c34c58b2-eb0e-4628-ba46-98207d6443ee" (UID: "c34c58b2-eb0e-4628-ba46-98207d6443ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:34.281893 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.281875 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34c58b2-eb0e-4628-ba46-98207d6443ee-kube-api-access-ht7ws" pod "c34c58b2-eb0e-4628-ba46-98207d6443ee" (UID: "c34c58b2-eb0e-4628-ba46-98207d6443ee"). InnerVolumeSpecName "kube-api-access-ht7ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:34.381714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.381615 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht7ws\" (UniqueName: \"kubernetes.io/projected/c34c58b2-eb0e-4628-ba46-98207d6443ee-kube-api-access-ht7ws\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:34.381714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.381652 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34c58b2-eb0e-4628-ba46-98207d6443ee-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:34.381714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.381662 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34c58b2-eb0e-4628-ba46-98207d6443ee-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:34.530383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.530345 4911 generic.go:356] "Generic (PLEG): container finished" podID="c34c58b2-eb0e-4628-ba46-98207d6443ee" containerID="1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570" exitCode=0 Nov 01 16:18:34.530755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.530712 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" event={"ID":"c34c58b2-eb0e-4628-ba46-98207d6443ee","Type":"ContainerDied","Data":"1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570"} Nov 01 16:18:34.530755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.530779 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" event={"ID":"c34c58b2-eb0e-4628-ba46-98207d6443ee","Type":"ContainerDied","Data":"172cc29edff5688e9af3f9580c2068bc28b3e0effcf8b1c68a53d5b9bbebb6c5"} Nov 01 16:18:34.530755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.530804 4911 scope.go:122] "RemoveContainer" containerID="1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570" Nov 01 16:18:34.531230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.530696 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh" Nov 01 16:18:34.544340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.544316 4911 scope.go:122] "RemoveContainer" containerID="1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570" Nov 01 16:18:34.544615 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:34.544593 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570\": container with ID starting with 1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570 not found: ID does not exist" containerID="1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570" Nov 01 16:18:34.544695 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.544623 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570"} err="failed to get container status \"1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570\": rpc error: code = NotFound desc = could not find container \"1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570\": container with ID starting with 1214023dc10f8cf6b244c1e99922b676bca1c930bf8e9a0e85a38b636fc01570 not found: ID does not exist" Nov 01 16:18:34.570148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.570107 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh"] Nov 01 16:18:34.576181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:34.576146 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-375350/test-model-375350-kserve-5ffdd99597-s6bmh"] Nov 01 16:18:35.970550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:35.970501 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34c58b2-eb0e-4628-ba46-98207d6443ee" path="/var/lib/kubelet/pods/c34c58b2-eb0e-4628-ba46-98207d6443ee/volumes" Nov 01 16:18:38.374852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:38.374808 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst"] Nov 01 16:18:38.375415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:38.375081 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" podUID="dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" containerName="maas-api" containerID="cri-o://835b5063fa48b47c0b34d3d910560ca7dd572be0fecda65fe754281da3fb3c72" gracePeriod=30 Nov 01 16:18:38.946290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:38.945159 4911 generic.go:356] "Generic (PLEG): container finished" podID="dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" containerID="835b5063fa48b47c0b34d3d910560ca7dd572be0fecda65fe754281da3fb3c72" exitCode=0 Nov 01 16:18:38.946290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:38.945452 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" event={"ID":"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a","Type":"ContainerDied","Data":"835b5063fa48b47c0b34d3d910560ca7dd572be0fecda65fe754281da3fb3c72"} Nov 01 16:18:39.083509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.083474 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:18:39.189478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.189443 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-kube-api-access-znxf5\" (UniqueName: \"kubernetes.io/projected/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-kube-api-access-znxf5\") pod \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " Nov 01 16:18:39.189890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.189861 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-metrics-certs\") pod \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " Nov 01 16:18:39.190022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.189931 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-openshift-service-ca\") pod \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " Nov 01 16:18:39.190157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.190093 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls\") pod \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\" (UID: \"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a\") " Nov 01 16:18:39.190527 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.190495 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-openshift-service-ca" pod "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" (UID: "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:39.190958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.190934 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:39.192506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.192476 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-kube-api-access-znxf5" pod "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" (UID: "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a"). InnerVolumeSpecName "kube-api-access-znxf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:39.192877 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.192846 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-metrics-certs" pod "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" (UID: "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:39.192877 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.192864 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls" pod "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" (UID: "dc407add-a1cd-4b18-b06c-f5f3c3c62f6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:39.216636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.216600 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" podUID="0f02b673-119f-412d-85db-d0bd27362550" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:18:39.216769 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.216712 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" podUID="3097fcd8-308a-4974-86f8-28bf25d8ee17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:18:39.293829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.293788 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxf5\" (UniqueName: \"kubernetes.io/projected/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-kube-api-access-znxf5\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:39.293829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.293830 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:39.294118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.293846 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:39.955879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.955844 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" Nov 01 16:18:39.956403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.955891 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst" event={"ID":"dc407add-a1cd-4b18-b06c-f5f3c3c62f6a","Type":"ContainerDied","Data":"4ba75d3def95a2404a077c3ceceb41209b22ce5074529667f6bf1adb3c4292ec"} Nov 01 16:18:39.956403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.955964 4911 scope.go:122] "RemoveContainer" containerID="835b5063fa48b47c0b34d3d910560ca7dd572be0fecda65fe754281da3fb3c72" Nov 01 16:18:39.993155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.993124 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst"] Nov 01 16:18:39.998815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:39.998787 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5dd93-5cb7b86db9-ksqst"] Nov 01 16:18:40.887392 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:40.887357 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:18:41.971854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:41.971759 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" path="/var/lib/kubelet/pods/dc407add-a1cd-4b18-b06c-f5f3c3c62f6a/volumes" Nov 01 16:18:49.215998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:49.215950 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" podUID="0f02b673-119f-412d-85db-d0bd27362550" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:18:49.215998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:49.215973 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" podUID="3097fcd8-308a-4974-86f8-28bf25d8ee17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:18:54.406858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.406817 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:18:54.497576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.496915 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:18:54.513061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.512998 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3097fcd8-308a-4974-86f8-28bf25d8ee17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3097fcd8-308a-4974-86f8-28bf25d8ee17-plugins-config-volume\") pod \"3097fcd8-308a-4974-86f8-28bf25d8ee17\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " Nov 01 16:18:54.513419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.513395 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3097fcd8-308a-4974-86f8-28bf25d8ee17-kube-api-access-92h2x\" (UniqueName: \"kubernetes.io/projected/3097fcd8-308a-4974-86f8-28bf25d8ee17-kube-api-access-92h2x\") pod \"3097fcd8-308a-4974-86f8-28bf25d8ee17\" (UID: \"3097fcd8-308a-4974-86f8-28bf25d8ee17\") " Nov 01 16:18:54.513625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.513599 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3097fcd8-308a-4974-86f8-28bf25d8ee17-plugins-config-volume" pod "3097fcd8-308a-4974-86f8-28bf25d8ee17" (UID: "3097fcd8-308a-4974-86f8-28bf25d8ee17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:54.514151 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.514126 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3097fcd8-308a-4974-86f8-28bf25d8ee17-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:54.516181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.516153 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3097fcd8-308a-4974-86f8-28bf25d8ee17-kube-api-access-92h2x" pod "3097fcd8-308a-4974-86f8-28bf25d8ee17" (UID: "3097fcd8-308a-4974-86f8-28bf25d8ee17"). InnerVolumeSpecName "kube-api-access-92h2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:54.615550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.615451 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f02b673-119f-412d-85db-d0bd27362550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f02b673-119f-412d-85db-d0bd27362550-plugins-config-volume\") pod \"0f02b673-119f-412d-85db-d0bd27362550\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " Nov 01 16:18:54.615808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.615795 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f02b673-119f-412d-85db-d0bd27362550-kube-api-access-rp8q4\" (UniqueName: \"kubernetes.io/projected/0f02b673-119f-412d-85db-d0bd27362550-kube-api-access-rp8q4\") pod \"0f02b673-119f-412d-85db-d0bd27362550\" (UID: \"0f02b673-119f-412d-85db-d0bd27362550\") " Nov 01 16:18:54.615852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.615820 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f02b673-119f-412d-85db-d0bd27362550-plugins-config-volume" pod "0f02b673-119f-412d-85db-d0bd27362550" (UID: "0f02b673-119f-412d-85db-d0bd27362550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:54.616218 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.616201 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f02b673-119f-412d-85db-d0bd27362550-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:54.616218 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.616217 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92h2x\" (UniqueName: \"kubernetes.io/projected/3097fcd8-308a-4974-86f8-28bf25d8ee17-kube-api-access-92h2x\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:54.617950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.617924 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f02b673-119f-412d-85db-d0bd27362550-kube-api-access-rp8q4" pod "0f02b673-119f-412d-85db-d0bd27362550" (UID: "0f02b673-119f-412d-85db-d0bd27362550"). InnerVolumeSpecName "kube-api-access-rp8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:54.717511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:54.717480 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8q4\" (UniqueName: \"kubernetes.io/projected/0f02b673-119f-412d-85db-d0bd27362550-kube-api-access-rp8q4\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:55.118238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.118199 4911 generic.go:356] "Generic (PLEG): container finished" podID="0f02b673-119f-412d-85db-d0bd27362550" containerID="8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f" exitCode=0 Nov 01 16:18:55.118433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.118258 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" Nov 01 16:18:55.118433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.118291 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" event={"ID":"0f02b673-119f-412d-85db-d0bd27362550","Type":"ContainerDied","Data":"8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f"} Nov 01 16:18:55.118433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.118363 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq" event={"ID":"0f02b673-119f-412d-85db-d0bd27362550","Type":"ContainerDied","Data":"9db5ac039a7262fbf2d1191c302494115ff43dd85de8e4c2979f20265f1eb09a"} Nov 01 16:18:55.118433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.118395 4911 scope.go:122] "RemoveContainer" containerID="8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f" Nov 01 16:18:55.120369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.120344 4911 generic.go:356] "Generic (PLEG): container finished" podID="3097fcd8-308a-4974-86f8-28bf25d8ee17" containerID="7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7" exitCode=0 Nov 01 16:18:55.120506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.120387 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" Nov 01 16:18:55.120506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.120423 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" event={"ID":"3097fcd8-308a-4974-86f8-28bf25d8ee17","Type":"ContainerDied","Data":"7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7"} Nov 01 16:18:55.120506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.120477 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc" event={"ID":"3097fcd8-308a-4974-86f8-28bf25d8ee17","Type":"ContainerDied","Data":"559f7ac0d4c82fda4c7bc279973f2f4be67a1e8d0ab59f2cb33a2af22dfba43e"} Nov 01 16:18:55.131864 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.131845 4911 scope.go:122] "RemoveContainer" containerID="8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f" Nov 01 16:18:55.132194 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:55.132177 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f\": container with ID starting with 8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f not found: ID does not exist" containerID="8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f" Nov 01 16:18:55.132247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.132202 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f"} err="failed to get container status \"8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f\": rpc error: code = NotFound desc = could not find container \"8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f\": container with ID starting with 8194be5be8ec6ff0301188b378af11899d1d286a8617471b4ed6dfc3b03b250f not found: ID does not exist" Nov 01 16:18:55.132247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.132218 4911 scope.go:122] "RemoveContainer" containerID="7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7" Nov 01 16:18:55.146220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.146181 4911 scope.go:122] "RemoveContainer" containerID="7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7" Nov 01 16:18:55.146598 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:18:55.146569 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7\": container with ID starting with 7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7 not found: ID does not exist" containerID="7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7" Nov 01 16:18:55.146697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.146608 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7"} err="failed to get container status \"7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7\": rpc error: code = NotFound desc = could not find container \"7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7\": container with ID starting with 7c8c415b6351bb6d69f7e920da5c3528f41116d50e91c80248634742e42abed7 not found: ID does not exist" Nov 01 16:18:55.147000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.146981 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq"] Nov 01 16:18:55.152113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.152087 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5dd93-5c967b5ff5-8gksq"] Nov 01 16:18:55.166312 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.166101 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc"] Nov 01 16:18:55.171257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:55.171233 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5dd93-85fd67b947-4wgxc"] Nov 01 16:18:56.020085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:56.020029 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f02b673-119f-412d-85db-d0bd27362550" path="/var/lib/kubelet/pods/0f02b673-119f-412d-85db-d0bd27362550/volumes" Nov 01 16:18:56.020687 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:18:56.020659 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3097fcd8-308a-4974-86f8-28bf25d8ee17" path="/var/lib/kubelet/pods/3097fcd8-308a-4974-86f8-28bf25d8ee17/volumes" Nov 01 16:19:06.438095 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:06.437838 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst"] Nov 01 16:19:06.438678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:06.438175 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" podUID="25db50cf-d326-4ea3-8bd7-c52af1c7bf86" containerName="maas-api" containerID="cri-o://9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0" gracePeriod=30 Nov 01 16:19:07.089951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.089919 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:19:07.249070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.245936 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\") pod \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " Nov 01 16:19:07.249070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.246113 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-metrics-certs\") pod \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " Nov 01 16:19:07.255013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.254171 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-kube-api-access-cxvdc\" (UniqueName: \"kubernetes.io/projected/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-kube-api-access-cxvdc\") pod \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " Nov 01 16:19:07.255013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.254398 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-openshift-service-ca\") pod \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\" (UID: \"25db50cf-d326-4ea3-8bd7-c52af1c7bf86\") " Nov 01 16:19:07.261011 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.256270 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-metrics-certs" pod "25db50cf-d326-4ea3-8bd7-c52af1c7bf86" (UID: "25db50cf-d326-4ea3-8bd7-c52af1c7bf86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:07.261011 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.256655 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-openshift-service-ca" pod "25db50cf-d326-4ea3-8bd7-c52af1c7bf86" (UID: "25db50cf-d326-4ea3-8bd7-c52af1c7bf86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:07.265392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.265231 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-kube-api-access-cxvdc" pod "25db50cf-d326-4ea3-8bd7-c52af1c7bf86" (UID: "25db50cf-d326-4ea3-8bd7-c52af1c7bf86"). InnerVolumeSpecName "kube-api-access-cxvdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:07.266939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.266906 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvdc\" (UniqueName: \"kubernetes.io/projected/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-kube-api-access-cxvdc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:07.266939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.266941 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:07.267215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.266958 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:07.268134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.268099 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls" pod "25db50cf-d326-4ea3-8bd7-c52af1c7bf86" (UID: "25db50cf-d326-4ea3-8bd7-c52af1c7bf86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:07.282343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.282307 4911 generic.go:356] "Generic (PLEG): container finished" podID="25db50cf-d326-4ea3-8bd7-c52af1c7bf86" containerID="9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0" exitCode=0 Nov 01 16:19:07.282478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.282374 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" Nov 01 16:19:07.282531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.282505 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" event={"ID":"25db50cf-d326-4ea3-8bd7-c52af1c7bf86","Type":"ContainerDied","Data":"9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0"} Nov 01 16:19:07.282568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.282557 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst" event={"ID":"25db50cf-d326-4ea3-8bd7-c52af1c7bf86","Type":"ContainerDied","Data":"ea81f63ff944a4c66db8be95ff36eabce9f5f0e16719fbba802211aafeea5041"} Nov 01 16:19:07.282609 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.282586 4911 scope.go:122] "RemoveContainer" containerID="9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0" Nov 01 16:19:07.309293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.309262 4911 scope.go:122] "RemoveContainer" containerID="9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0" Nov 01 16:19:07.309748 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:07.309699 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0\": container with ID starting with 9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0 not found: ID does not exist" containerID="9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0" Nov 01 16:19:07.309993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.309958 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0"} err="failed to get container status \"9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0\": rpc error: code = NotFound desc = could not find container \"9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0\": container with ID starting with 9d9118a64a173008ec7f0dbf558b235cfbc59343de698133e14ea9e94fa16ec0 not found: ID does not exist" Nov 01 16:19:07.321034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.321000 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst"] Nov 01 16:19:07.325600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.325571 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c65e3-6cdfdc4477-g5jst"] Nov 01 16:19:07.368873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.368830 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25db50cf-d326-4ea3-8bd7-c52af1c7bf86-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:07.970939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:07.970646 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25db50cf-d326-4ea3-8bd7-c52af1c7bf86" path="/var/lib/kubelet/pods/25db50cf-d326-4ea3-8bd7-c52af1c7bf86/volumes" Nov 01 16:19:09.513530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:09.513490 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d"] Nov 01 16:19:10.868084 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:10.868028 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:19:15.459064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:15.456602 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc"] Nov 01 16:19:15.459064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:15.456916 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" containerID="cri-o://7dbe62465e999ec72d5210b8cefa2ea1f7211af55bbf98517570148abe07946e" gracePeriod=30 Nov 01 16:19:15.860629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:15.860591 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8"] Nov 01 16:19:15.860849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:15.860811 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" containerID="cri-o://66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104" gracePeriod=30 Nov 01 16:19:16.166110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:16.166004 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq"] Nov 01 16:19:16.166345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:16.166312 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" containerID="cri-o://32c88a35dff80ca9776af9bcfc891e313199d75534c025b78b81198f9bd4891a" gracePeriod=30 Nov 01 16:19:18.846763 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:18.846723 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj"] Nov 01 16:19:18.847203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:18.846950 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" containerID="cri-o://cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9" gracePeriod=30 Nov 01 16:19:20.377059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:20.376931 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8"] Nov 01 16:19:20.621100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:20.617754 4911 generic.go:356] "Generic (PLEG): container finished" podID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerID="7dbe62465e999ec72d5210b8cefa2ea1f7211af55bbf98517570148abe07946e" exitCode=0 Nov 01 16:19:20.621100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:20.618030 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" event={"ID":"6f3ab62e-a9b7-4485-87f6-aa6f3519d446","Type":"ContainerDied","Data":"7dbe62465e999ec72d5210b8cefa2ea1f7211af55bbf98517570148abe07946e"} Nov 01 16:19:21.327166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.327134 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:19:21.332938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.332903 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:19:21.437583 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437543 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-certs\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437660 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db969f4-c84d-47f7-8240-130d785859c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db969f4-c84d-47f7-8240-130d785859c1-istiod-ca-cert\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437692 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istiod-ca-cert\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437773 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-kube-api-access-8xvzx\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-kube-api-access-8xvzx\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437840 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-token\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-token\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437874 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-socket\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437924 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-socket\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.437927 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-certs" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438012 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-envoy\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438066 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-credential-socket\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438129 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db969f4-c84d-47f7-8240-130d785859c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db969f4-c84d-47f7-8240-130d785859c1-istio-podinfo\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438215 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-envoy\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438265 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-podinfo\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438314 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-credential-socket\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438355 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-istio-token\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-istio-token\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438471 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-data\") pod \"5db969f4-c84d-47f7-8240-130d785859c1\" (UID: \"5db969f4-c84d-47f7-8240-130d785859c1\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438505 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-certs\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438538 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-data\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.438626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.438592 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-kube-api-access-fd4zk\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-kube-api-access-fd4zk\") pod \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\" (UID: \"6f3ab62e-a9b7-4485-87f6-aa6f3519d446\") " Nov 01 16:19:21.439475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.439447 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.440197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.440168 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db969f4-c84d-47f7-8240-130d785859c1-istiod-ca-cert" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.440591 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.440566 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istiod-ca-cert" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.441121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.441094 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-certs" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.441307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.441287 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-data" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.441541 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.441516 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-credential-socket" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.441775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.441726 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-data" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.441948 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.441927 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-socket" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.442662 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.442634 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-kube-api-access-8xvzx" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "kube-api-access-8xvzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.442863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.442837 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-envoy" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.443245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.442867 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-credential-socket" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.443388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.443364 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-kube-api-access-fd4zk" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "kube-api-access-fd4zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.443865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.443836 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-socket" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.444543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.444513 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-token" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.445257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.445229 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-istio-token" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.445985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.445926 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-envoy" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.448229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.448150 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db969f4-c84d-47f7-8240-130d785859c1-istio-podinfo" pod "5db969f4-c84d-47f7-8240-130d785859c1" (UID: "5db969f4-c84d-47f7-8240-130d785859c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:21.448639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.448586 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-podinfo" pod "6f3ab62e-a9b7-4485-87f6-aa6f3519d446" (UID: "6f3ab62e-a9b7-4485-87f6-aa6f3519d446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:21.548126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.548092 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.548544 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.548521 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.548705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.548681 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.548827 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.548813 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549138 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549275 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db969f4-c84d-47f7-8240-130d785859c1-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549297 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549312 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549542 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549555 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549569 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db969f4-c84d-47f7-8240-130d785859c1-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549595 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549607 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549621 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd4zk\" (UniqueName: \"kubernetes.io/projected/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-kube-api-access-fd4zk\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549635 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db969f4-c84d-47f7-8240-130d785859c1-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549644 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3ab62e-a9b7-4485-87f6-aa6f3519d446-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.549669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.549652 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvzx\" (UniqueName: \"kubernetes.io/projected/5db969f4-c84d-47f7-8240-130d785859c1-kube-api-access-8xvzx\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.633884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.633846 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" event={"ID":"6f3ab62e-a9b7-4485-87f6-aa6f3519d446","Type":"ContainerDied","Data":"6ca56b3abdc18ad9d53ebac1106e8e1dfdc665a9788e5adb5487c7f129015846"} Nov 01 16:19:21.634069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.633924 4911 scope.go:122] "RemoveContainer" containerID="7dbe62465e999ec72d5210b8cefa2ea1f7211af55bbf98517570148abe07946e" Nov 01 16:19:21.634163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.634146 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc" Nov 01 16:19:21.643067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.642399 4911 generic.go:356] "Generic (PLEG): container finished" podID="5db969f4-c84d-47f7-8240-130d785859c1" containerID="66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104" exitCode=0 Nov 01 16:19:21.643067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.642518 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" event={"ID":"5db969f4-c84d-47f7-8240-130d785859c1","Type":"ContainerDied","Data":"66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104"} Nov 01 16:19:21.643067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.642565 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" event={"ID":"5db969f4-c84d-47f7-8240-130d785859c1","Type":"ContainerDied","Data":"1b9c619b699288a4ef80531cae5dd5c25761e60432076db9984ed91de7be3bff"} Nov 01 16:19:21.643067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.642641 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8" Nov 01 16:19:21.647782 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.647480 4911 generic.go:356] "Generic (PLEG): container finished" podID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerID="32c88a35dff80ca9776af9bcfc891e313199d75534c025b78b81198f9bd4891a" exitCode=0 Nov 01 16:19:21.647782 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.647529 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" event={"ID":"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a","Type":"ContainerDied","Data":"32c88a35dff80ca9776af9bcfc891e313199d75534c025b78b81198f9bd4891a"} Nov 01 16:19:21.647782 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.647570 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" event={"ID":"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a","Type":"ContainerDied","Data":"befc2266915292da75ece468d0deee05ebe1a4b4c46cf8c5fd4015bfee63e099"} Nov 01 16:19:21.647782 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.647591 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="befc2266915292da75ece468d0deee05ebe1a4b4c46cf8c5fd4015bfee63e099" Nov 01 16:19:21.667290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.667264 4911 scope.go:122] "RemoveContainer" containerID="66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104" Nov 01 16:19:21.669285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.669242 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:19:21.691631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.691604 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc"] Nov 01 16:19:21.691720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.691685 4911 scope.go:122] "RemoveContainer" containerID="66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104" Nov 01 16:19:21.692165 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:21.692123 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104\": container with ID starting with 66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104 not found: ID does not exist" containerID="66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104" Nov 01 16:19:21.692265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.692163 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104"} err="failed to get container status \"66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104\": rpc error: code = NotFound desc = could not find container \"66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104\": container with ID starting with 66d7ee325b5811620e87187ea91155c2735ff88e11656a43efcab16561f5e104 not found: ID does not exist" Nov 01 16:19:21.698740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.698667 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-k7wzc"] Nov 01 16:19:21.706646 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.706604 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8"] Nov 01 16:19:21.711426 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.711397 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-9gnt8"] Nov 01 16:19:21.751563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.751519 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-credential-socket\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.751740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.751642 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-token\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-token\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.751821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.751744 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-podinfo\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.751884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.751834 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-kube-api-access-drtbf\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-kube-api-access-drtbf\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.751955 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.751938 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-data\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.752114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.752097 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-certs\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.752177 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.752140 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-socket\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.752231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.752218 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-envoy\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.752282 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.752254 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istiod-ca-cert\") pod \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\" (UID: \"b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a\") " Nov 01 16:19:21.753462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.753359 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istiod-ca-cert" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.753639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.753621 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-credential-socket" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.754139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.754117 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-data" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.759104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.756587 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-podinfo" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:21.759104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.756909 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-certs" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.759104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.757350 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-socket" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.759104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.757755 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-kube-api-access-drtbf" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "kube-api-access-drtbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.759104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.758400 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-token" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.759491 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.759464 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-envoy" pod "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" (UID: "b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.854531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854492 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854528 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drtbf\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-kube-api-access-drtbf\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854542 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854556 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854568 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854580 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854591 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854603 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.854794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.854615 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.973729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.973633 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db969f4-c84d-47f7-8240-130d785859c1" path="/var/lib/kubelet/pods/5db969f4-c84d-47f7-8240-130d785859c1/volumes" Nov 01 16:19:21.974676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:21.974647 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" path="/var/lib/kubelet/pods/6f3ab62e-a9b7-4485-87f6-aa6f3519d446/volumes" Nov 01 16:19:22.659798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:22.659759 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq" Nov 01 16:19:22.681963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:22.681931 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq"] Nov 01 16:19:22.690798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:22.690768 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-664vq"] Nov 01 16:19:23.970956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:23.970920 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" path="/var/lib/kubelet/pods/b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a/volumes" Nov 01 16:19:24.191689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.191661 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.343675 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-token\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-token\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.343807 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-socket\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.343852 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-data\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.343891 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-podinfo\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.343933 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-credential-socket\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.343977 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-kube-api-access-btjm2\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-kube-api-access-btjm2\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.344340 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-certs\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.344412 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3955395d-d20b-4df9-ae0f-8da5715e29b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955395d-d20b-4df9-ae0f-8da5715e29b3-istiod-ca-cert\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.346115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.344441 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-envoy\") pod \"3955395d-d20b-4df9-ae0f-8da5715e29b3\" (UID: \"3955395d-d20b-4df9-ae0f-8da5715e29b3\") " Nov 01 16:19:24.398304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.390229 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-socket" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.398304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.390660 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-data" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.403414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.402597 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-credential-socket" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.409679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.404028 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-token" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:24.409679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.408447 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3955395d-d20b-4df9-ae0f-8da5715e29b3-istiod-ca-cert" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:24.410509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.410466 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-podinfo" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:24.415266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.415204 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-certs" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.430075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.423460 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-kube-api-access-btjm2" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "kube-api-access-btjm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:24.438072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.437146 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-envoy" pod "3955395d-d20b-4df9-ae0f-8da5715e29b3" (UID: "3955395d-d20b-4df9-ae0f-8da5715e29b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.447060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.446967 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447011 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955395d-d20b-4df9-ae0f-8da5715e29b3-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447026 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447364 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447384 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447398 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447411 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955395d-d20b-4df9-ae0f-8da5715e29b3-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447632 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955395d-d20b-4df9-ae0f-8da5715e29b3-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.447732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.447647 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btjm2\" (UniqueName: \"kubernetes.io/projected/3955395d-d20b-4df9-ae0f-8da5715e29b3-kube-api-access-btjm2\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.537913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.537864 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" podUID="77b69ff4-60ba-477a-b077-406f0f88b4d6" containerName="main" containerID="cri-o://0333d40d0f58b999b30702b866679ce82ac7bd3a1c0b759edee132285abb8b14" gracePeriod=45 Nov 01 16:19:24.683398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.683364 4911 generic.go:356] "Generic (PLEG): container finished" podID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerID="cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9" exitCode=0 Nov 01 16:19:24.683586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.683436 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" Nov 01 16:19:24.683586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.683447 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" event={"ID":"3955395d-d20b-4df9-ae0f-8da5715e29b3","Type":"ContainerDied","Data":"cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9"} Nov 01 16:19:24.683586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.683514 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj" event={"ID":"3955395d-d20b-4df9-ae0f-8da5715e29b3","Type":"ContainerDied","Data":"fcca20f5b04652b3d23d69a4d185d7d46f228d922d6225193cff9d501389dc59"} Nov 01 16:19:24.683586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.683553 4911 scope.go:122] "RemoveContainer" containerID="cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9" Nov 01 16:19:24.691628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.691592 4911 generic.go:356] "Generic (PLEG): container finished" podID="77b69ff4-60ba-477a-b077-406f0f88b4d6" containerID="0333d40d0f58b999b30702b866679ce82ac7bd3a1c0b759edee132285abb8b14" exitCode=0 Nov 01 16:19:24.691772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.691668 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" event={"ID":"77b69ff4-60ba-477a-b077-406f0f88b4d6","Type":"ContainerDied","Data":"0333d40d0f58b999b30702b866679ce82ac7bd3a1c0b759edee132285abb8b14"} Nov 01 16:19:24.704724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.704698 4911 scope.go:122] "RemoveContainer" containerID="cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9" Nov 01 16:19:24.705098 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:24.705072 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9\": container with ID starting with cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9 not found: ID does not exist" containerID="cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9" Nov 01 16:19:24.705203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.705105 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9"} err="failed to get container status \"cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9\": rpc error: code = NotFound desc = could not find container \"cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9\": container with ID starting with cf0426f0dd86dceaac01d6ba7e3f4946e6ec68cde2aed412fb17fba99a8052c9 not found: ID does not exist" Nov 01 16:19:24.727174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.727066 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj"] Nov 01 16:19:24.752790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.737644 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-whjkj"] Nov 01 16:19:24.840919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.840886 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:19:24.957113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.956210 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-dshm\") pod \"77b69ff4-60ba-477a-b077-406f0f88b4d6\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " Nov 01 16:19:24.957113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.956444 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b69ff4-60ba-477a-b077-406f0f88b4d6-kube-api-access-ftxm9\" (UniqueName: \"kubernetes.io/projected/77b69ff4-60ba-477a-b077-406f0f88b4d6-kube-api-access-ftxm9\") pod \"77b69ff4-60ba-477a-b077-406f0f88b4d6\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " Nov 01 16:19:24.957113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.956486 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-model-cache\") pod \"77b69ff4-60ba-477a-b077-406f0f88b4d6\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " Nov 01 16:19:24.957113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.956590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-tmp-dir\") pod \"77b69ff4-60ba-477a-b077-406f0f88b4d6\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " Nov 01 16:19:24.957113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.956711 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77b69ff4-60ba-477a-b077-406f0f88b4d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/77b69ff4-60ba-477a-b077-406f0f88b4d6-tls-certs\") pod \"77b69ff4-60ba-477a-b077-406f0f88b4d6\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " Nov 01 16:19:24.957113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.956756 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-home\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-home\") pod \"77b69ff4-60ba-477a-b077-406f0f88b4d6\" (UID: \"77b69ff4-60ba-477a-b077-406f0f88b4d6\") " Nov 01 16:19:24.960325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.957704 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-home" pod "77b69ff4-60ba-477a-b077-406f0f88b4d6" (UID: "77b69ff4-60ba-477a-b077-406f0f88b4d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.960325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.959279 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-model-cache" pod "77b69ff4-60ba-477a-b077-406f0f88b4d6" (UID: "77b69ff4-60ba-477a-b077-406f0f88b4d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.960325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.959931 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-tmp-dir" pod "77b69ff4-60ba-477a-b077-406f0f88b4d6" (UID: "77b69ff4-60ba-477a-b077-406f0f88b4d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.962698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.962675 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b69ff4-60ba-477a-b077-406f0f88b4d6-kube-api-access-ftxm9" pod "77b69ff4-60ba-477a-b077-406f0f88b4d6" (UID: "77b69ff4-60ba-477a-b077-406f0f88b4d6"). InnerVolumeSpecName "kube-api-access-ftxm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:24.962817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.962790 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-dshm" pod "77b69ff4-60ba-477a-b077-406f0f88b4d6" (UID: "77b69ff4-60ba-477a-b077-406f0f88b4d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:24.966187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:24.964096 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b69ff4-60ba-477a-b077-406f0f88b4d6-tls-certs" pod "77b69ff4-60ba-477a-b077-406f0f88b4d6" (UID: "77b69ff4-60ba-477a-b077-406f0f88b4d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:25.060663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.060628 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b69ff4-60ba-477a-b077-406f0f88b4d6-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:25.060663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.060660 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:25.060663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.060673 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:25.061356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.060683 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxm9\" (UniqueName: \"kubernetes.io/projected/77b69ff4-60ba-477a-b077-406f0f88b4d6-kube-api-access-ftxm9\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:25.061356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.060692 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:25.061356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.060703 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b69ff4-60ba-477a-b077-406f0f88b4d6-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:25.701338 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.701300 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" event={"ID":"77b69ff4-60ba-477a-b077-406f0f88b4d6","Type":"ContainerDied","Data":"d7734c4a21c962cb7ac38fa5f5770fb6fd36bc1fa9c1dc34f610e5a52543bb2e"} Nov 01 16:19:25.701530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.701317 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d" Nov 01 16:19:25.701530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.701366 4911 scope.go:122] "RemoveContainer" containerID="0333d40d0f58b999b30702b866679ce82ac7bd3a1c0b759edee132285abb8b14" Nov 01 16:19:25.727229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.727200 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d"] Nov 01 16:19:25.731811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.731778 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2001b0/test-model-2001b0-kserve-7645fb85fc-mg82d"] Nov 01 16:19:25.969617 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.969541 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" path="/var/lib/kubelet/pods/3955395d-d20b-4df9-ae0f-8da5715e29b3/volumes" Nov 01 16:19:25.970218 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:25.970200 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b69ff4-60ba-477a-b077-406f0f88b4d6" path="/var/lib/kubelet/pods/77b69ff4-60ba-477a-b077-406f0f88b4d6/volumes" Nov 01 16:19:35.406331 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.406282 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" podUID="f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" containerName="main" containerID="cri-o://d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29" gracePeriod=45 Nov 01 16:19:35.709278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.709242 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:19:35.784889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.784855 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-dshm\") pod \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " Nov 01 16:19:35.785093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.784946 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-kube-api-access-64vzr\" (UniqueName: \"kubernetes.io/projected/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-kube-api-access-64vzr\") pod \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " Nov 01 16:19:35.785221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.785198 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tls-certs\") pod \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " Nov 01 16:19:35.785285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.785257 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tmp-dir\") pod \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " Nov 01 16:19:35.785392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.785378 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-home\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-home\") pod \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " Nov 01 16:19:35.785461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.785452 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-model-cache\") pod \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\" (UID: \"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f\") " Nov 01 16:19:35.785719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.785694 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tmp-dir" pod "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" (UID: "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:35.785804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.785767 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-home" pod "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" (UID: "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:35.785911 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.785892 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-model-cache" pod "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" (UID: "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:35.786360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.786331 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:35.786453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.786364 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:35.786453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.786379 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:35.787914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.787881 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-dshm" pod "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" (UID: "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:35.788324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.788295 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-kube-api-access-64vzr" pod "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" (UID: "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f"). InnerVolumeSpecName "kube-api-access-64vzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:35.788413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.788306 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tls-certs" pod "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" (UID: "f4aa6807-c041-4c0e-b2ab-8bcbbc69033f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:35.811247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.811216 4911 generic.go:356] "Generic (PLEG): container finished" podID="f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" containerID="d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29" exitCode=0 Nov 01 16:19:35.811419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.811259 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" Nov 01 16:19:35.811419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.811299 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" event={"ID":"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f","Type":"ContainerDied","Data":"d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29"} Nov 01 16:19:35.811419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.811342 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8" event={"ID":"f4aa6807-c041-4c0e-b2ab-8bcbbc69033f","Type":"ContainerDied","Data":"9c4f163f99607ddf79e9d9d2a47dbc4ee91b225b38ae4f6252965eddcb475bc3"} Nov 01 16:19:35.811419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.811363 4911 scope.go:122] "RemoveContainer" containerID="d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29" Nov 01 16:19:35.825693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.825670 4911 scope.go:122] "RemoveContainer" containerID="d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29" Nov 01 16:19:35.825996 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:35.825978 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29\": container with ID starting with d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29 not found: ID does not exist" containerID="d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29" Nov 01 16:19:35.826076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.826006 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29"} err="failed to get container status \"d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29\": rpc error: code = NotFound desc = could not find container \"d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29\": container with ID starting with d1e1d2e5daa56533ec18a1c56432fbd442adf9a90c87a4c0b715423a2a802b29 not found: ID does not exist" Nov 01 16:19:35.838544 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.838516 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8"] Nov 01 16:19:35.843081 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.843055 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-99afca/test-model-99afca-kserve-58f55d5b69-qkkn8"] Nov 01 16:19:35.888349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.888313 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vzr\" (UniqueName: \"kubernetes.io/projected/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-kube-api-access-64vzr\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:35.888349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.888344 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:35.888349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.888355 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:35.974808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:35.974724 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" path="/var/lib/kubelet/pods/f4aa6807-c041-4c0e-b2ab-8bcbbc69033f/volumes" Nov 01 16:19:36.399158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.399122 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw"] Nov 01 16:19:36.399414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.399387 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" podUID="c254b441-da43-44c6-9bb7-6fa6775b4514" containerName="maas-api" containerID="cri-o://6eddc5d6e48ea29994890039e499804ca6521ee1c29eff01b12030c146a19b72" gracePeriod=30 Nov 01 16:19:36.577909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.577867 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl"] Nov 01 16:19:36.578423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.578132 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" podUID="4020e3b4-ec83-4429-89a8-467c965a5e5f" containerName="maas-api" containerID="cri-o://24583d522f8b7251aef40aec5b8fbe974b2cac5e492230e41cdd1de8b92c0b6a" gracePeriod=30 Nov 01 16:19:36.827693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.827473 4911 generic.go:356] "Generic (PLEG): container finished" podID="c254b441-da43-44c6-9bb7-6fa6775b4514" containerID="6eddc5d6e48ea29994890039e499804ca6521ee1c29eff01b12030c146a19b72" exitCode=0 Nov 01 16:19:36.827693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.827607 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" event={"ID":"c254b441-da43-44c6-9bb7-6fa6775b4514","Type":"ContainerDied","Data":"6eddc5d6e48ea29994890039e499804ca6521ee1c29eff01b12030c146a19b72"} Nov 01 16:19:36.831070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.830733 4911 generic.go:356] "Generic (PLEG): container finished" podID="4020e3b4-ec83-4429-89a8-467c965a5e5f" containerID="24583d522f8b7251aef40aec5b8fbe974b2cac5e492230e41cdd1de8b92c0b6a" exitCode=0 Nov 01 16:19:36.831070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:36.830874 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" event={"ID":"4020e3b4-ec83-4429-89a8-467c965a5e5f","Type":"ContainerDied","Data":"24583d522f8b7251aef40aec5b8fbe974b2cac5e492230e41cdd1de8b92c0b6a"} Nov 01 16:19:37.172631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.172244 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:19:37.179881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.179182 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:19:37.275617 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.275572 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4020e3b4-ec83-4429-89a8-467c965a5e5f-kube-api-access-4qv26\" (UniqueName: \"kubernetes.io/projected/4020e3b4-ec83-4429-89a8-467c965a5e5f-kube-api-access-4qv26\") pod \"4020e3b4-ec83-4429-89a8-467c965a5e5f\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " Nov 01 16:19:37.275877 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.275859 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4020e3b4-ec83-4429-89a8-467c965a5e5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4020e3b4-ec83-4429-89a8-467c965a5e5f-openshift-service-ca\") pod \"4020e3b4-ec83-4429-89a8-467c965a5e5f\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " Nov 01 16:19:37.276531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.276502 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c254b441-da43-44c6-9bb7-6fa6775b4514-openshift-service-ca" pod "c254b441-da43-44c6-9bb7-6fa6775b4514" (UID: "c254b441-da43-44c6-9bb7-6fa6775b4514"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:37.277006 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.276985 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c254b441-da43-44c6-9bb7-6fa6775b4514-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c254b441-da43-44c6-9bb7-6fa6775b4514-openshift-service-ca\") pod \"c254b441-da43-44c6-9bb7-6fa6775b4514\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " Nov 01 16:19:37.298620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.298590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c254b441-da43-44c6-9bb7-6fa6775b4514-kube-api-access-nv4dj\" (UniqueName: \"kubernetes.io/projected/c254b441-da43-44c6-9bb7-6fa6775b4514-kube-api-access-nv4dj\") pod \"c254b441-da43-44c6-9bb7-6fa6775b4514\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " Nov 01 16:19:37.301152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.299319 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-metrics-certs\") pod \"4020e3b4-ec83-4429-89a8-467c965a5e5f\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " Nov 01 16:19:37.316561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.296286 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4020e3b4-ec83-4429-89a8-467c965a5e5f-openshift-service-ca" pod "4020e3b4-ec83-4429-89a8-467c965a5e5f" (UID: "4020e3b4-ec83-4429-89a8-467c965a5e5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:37.317328 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.317302 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-metrics-certs\") pod \"c254b441-da43-44c6-9bb7-6fa6775b4514\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " Nov 01 16:19:37.317576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.317554 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-maas-api-tls\") pod \"4020e3b4-ec83-4429-89a8-467c965a5e5f\" (UID: \"4020e3b4-ec83-4429-89a8-467c965a5e5f\") " Nov 01 16:19:37.318179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.318161 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls\") pod \"c254b441-da43-44c6-9bb7-6fa6775b4514\" (UID: \"c254b441-da43-44c6-9bb7-6fa6775b4514\") " Nov 01 16:19:37.328858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.328770 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c254b441-da43-44c6-9bb7-6fa6775b4514-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.328858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.328807 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4020e3b4-ec83-4429-89a8-467c965a5e5f-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.392580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.392533 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-metrics-certs" pod "c254b441-da43-44c6-9bb7-6fa6775b4514" (UID: "c254b441-da43-44c6-9bb7-6fa6775b4514"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:37.395183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.393173 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c254b441-da43-44c6-9bb7-6fa6775b4514-kube-api-access-nv4dj" pod "c254b441-da43-44c6-9bb7-6fa6775b4514" (UID: "c254b441-da43-44c6-9bb7-6fa6775b4514"). InnerVolumeSpecName "kube-api-access-nv4dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:37.404760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.404594 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4020e3b4-ec83-4429-89a8-467c965a5e5f-kube-api-access-4qv26" pod "4020e3b4-ec83-4429-89a8-467c965a5e5f" (UID: "4020e3b4-ec83-4429-89a8-467c965a5e5f"). InnerVolumeSpecName "kube-api-access-4qv26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:37.410021 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.409968 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-maas-api-tls" pod "4020e3b4-ec83-4429-89a8-467c965a5e5f" (UID: "4020e3b4-ec83-4429-89a8-467c965a5e5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:37.414999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.414123 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-metrics-certs" pod "4020e3b4-ec83-4429-89a8-467c965a5e5f" (UID: "4020e3b4-ec83-4429-89a8-467c965a5e5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:37.445639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.444136 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qv26\" (UniqueName: \"kubernetes.io/projected/4020e3b4-ec83-4429-89a8-467c965a5e5f-kube-api-access-4qv26\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.445639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.444169 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4dj\" (UniqueName: \"kubernetes.io/projected/c254b441-da43-44c6-9bb7-6fa6775b4514-kube-api-access-nv4dj\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.445639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.444185 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.445639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.444205 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.445639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.444218 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4020e3b4-ec83-4429-89a8-467c965a5e5f-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.445639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.444604 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls" pod "c254b441-da43-44c6-9bb7-6fa6775b4514" (UID: "c254b441-da43-44c6-9bb7-6fa6775b4514"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:37.547030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.546920 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c254b441-da43-44c6-9bb7-6fa6775b4514-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:37.843406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.843309 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" Nov 01 16:19:37.843406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.843318 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw" event={"ID":"c254b441-da43-44c6-9bb7-6fa6775b4514","Type":"ContainerDied","Data":"ea00cc1fdc1c8d523d1e64920272e3c1b8e1c3b071a5731d6dae338d978af034"} Nov 01 16:19:37.843956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.843428 4911 scope.go:122] "RemoveContainer" containerID="6eddc5d6e48ea29994890039e499804ca6521ee1c29eff01b12030c146a19b72" Nov 01 16:19:37.850660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.845804 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" event={"ID":"4020e3b4-ec83-4429-89a8-467c965a5e5f","Type":"ContainerDied","Data":"adb13b7e3bcba6ad4ff963e46889609eae2095e33aa8fc4447d99c30e2a14549"} Nov 01 16:19:37.850660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.845817 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl" Nov 01 16:19:37.861440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.861412 4911 scope.go:122] "RemoveContainer" containerID="24583d522f8b7251aef40aec5b8fbe974b2cac5e492230e41cdd1de8b92c0b6a" Nov 01 16:19:37.896889 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.896855 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw"] Nov 01 16:19:37.901780 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.901752 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-784444-57f58cd57f-649bw"] Nov 01 16:19:37.916745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.913794 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl"] Nov 01 16:19:37.934427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.934389 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2ce08a-57c6db8dbb-jk2wl"] Nov 01 16:19:37.980871 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.980830 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4020e3b4-ec83-4429-89a8-467c965a5e5f" path="/var/lib/kubelet/pods/4020e3b4-ec83-4429-89a8-467c965a5e5f/volumes" Nov 01 16:19:37.981647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:37.981606 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c254b441-da43-44c6-9bb7-6fa6775b4514" path="/var/lib/kubelet/pods/c254b441-da43-44c6-9bb7-6fa6775b4514/volumes" Nov 01 16:19:39.334010 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.333971 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6"] Nov 01 16:19:39.335204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335177 4911 cpu_manager.go:414] "containerMap: removing container" podUID="3097fcd8-308a-4974-86f8-28bf25d8ee17" containerName="payload-processing" Nov 01 16:19:39.335293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335206 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3097fcd8-308a-4974-86f8-28bf25d8ee17" containerName="payload-processing" Nov 01 16:19:39.335293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335234 4911 cpu_manager.go:414] "containerMap: removing container" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" Nov 01 16:19:39.335293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335242 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" Nov 01 16:19:39.335293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335259 4911 cpu_manager.go:414] "containerMap: removing container" podUID="dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" containerName="maas-api" Nov 01 16:19:39.335293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335267 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" containerName="maas-api" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335297 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335305 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335318 4911 cpu_manager.go:414] "containerMap: removing container" podUID="c34c58b2-eb0e-4628-ba46-98207d6443ee" containerName="main" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335324 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34c58b2-eb0e-4628-ba46-98207d6443ee" containerName="main" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335363 4911 cpu_manager.go:414] "containerMap: removing container" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335371 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335381 4911 cpu_manager.go:414] "containerMap: removing container" podUID="77b69ff4-60ba-477a-b077-406f0f88b4d6" containerName="main" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335389 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b69ff4-60ba-477a-b077-406f0f88b4d6" containerName="main" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335408 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335415 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335430 4911 cpu_manager.go:414] "containerMap: removing container" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335437 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335451 4911 cpu_manager.go:414] "containerMap: removing container" podUID="0f02b673-119f-412d-85db-d0bd27362550" containerName="payload-pre-processing" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335460 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f02b673-119f-412d-85db-d0bd27362550" containerName="payload-pre-processing" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335471 4911 cpu_manager.go:414] "containerMap: removing container" podUID="25db50cf-d326-4ea3-8bd7-c52af1c7bf86" containerName="maas-api" Nov 01 16:19:39.335483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335478 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25db50cf-d326-4ea3-8bd7-c52af1c7bf86" containerName="maas-api" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335491 4911 cpu_manager.go:414] "containerMap: removing container" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335498 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335513 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" containerName="main" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335520 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" containerName="main" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335550 4911 cpu_manager.go:414] "containerMap: removing container" podUID="c254b441-da43-44c6-9bb7-6fa6775b4514" containerName="maas-api" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335558 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c254b441-da43-44c6-9bb7-6fa6775b4514" containerName="maas-api" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335587 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335595 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335616 4911 cpu_manager.go:414] "containerMap: removing container" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335622 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335654 4911 cpu_manager.go:414] "containerMap: removing container" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335661 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335675 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4020e3b4-ec83-4429-89a8-467c965a5e5f" containerName="maas-api" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335682 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4020e3b4-ec83-4429-89a8-467c965a5e5f" containerName="maas-api" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335875 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db969f4-c84d-47f7-8240-130d785859c1" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335887 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b69ff4-60ba-477a-b077-406f0f88b4d6" containerName="main" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335902 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ef0c7-b67d-4849-a0f1-b4b84aa022e3" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335915 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3ab62e-a9b7-4485-87f6-aa6f3519d446" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335927 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f02b673-119f-412d-85db-d0bd27362550" containerName="payload-pre-processing" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335937 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="25db50cf-d326-4ea3-8bd7-c52af1c7bf86" containerName="maas-api" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335948 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="662ec3c9-d7fb-4cbe-a1b8-c432eb7c96df" containerName="istio-proxy" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335956 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4aa6807-c041-4c0e-b2ab-8bcbbc69033f" containerName="main" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335968 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="c254b441-da43-44c6-9bb7-6fa6775b4514" containerName="maas-api" Nov 01 16:19:39.335971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335980 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50de64b-b0d0-4cd5-9b16-7c92d8d37f5a" containerName="istio-proxy" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.335990 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="60af24b8-930b-48af-a4d9-0ee884f97bb3" containerName="istio-proxy" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.336002 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="542c63ce-b126-4f1d-8eaf-d5ca3e087df6" containerName="istio-proxy" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.336012 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="4020e3b4-ec83-4429-89a8-467c965a5e5f" containerName="maas-api" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.336022 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="3097fcd8-308a-4974-86f8-28bf25d8ee17" containerName="payload-processing" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.336031 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="3955395d-d20b-4df9-ae0f-8da5715e29b3" containerName="istio-proxy" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.336056 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc407add-a1cd-4b18-b06c-f5f3c3c62f6a" containerName="maas-api" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.336067 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16c4f4e-8946-4ec7-a5d2-2d08c925dad1" containerName="istio-proxy" Nov 01 16:19:39.336818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.336077 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34c58b2-eb0e-4628-ba46-98207d6443ee" containerName="main" Nov 01 16:19:39.342907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.342881 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.352417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.352382 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6"] Nov 01 16:19:39.352891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.352831 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d121\"/\"openshift-service-ca.crt\"" Nov 01 16:19:39.352891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.352855 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d121\"/\"auth-test-model-c8d121-kserve-self-signed-certs\"" Nov 01 16:19:39.353171 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.352828 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d121\"/\"kube-root-ca.crt\"" Nov 01 16:19:39.353439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.353421 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d121\"/\"default-dockercfg-bvplb\"" Nov 01 16:19:39.383127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.381159 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jww\" (UniqueName: \"kubernetes.io/projected/fb7b581d-c541-401b-a814-81bdc2737087-kube-api-access-j8jww\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.383127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.381300 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-dshm\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.383127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.381383 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-model-cache\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.383127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.381412 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-tmp-dir\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.383127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.381538 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7b581d-c541-401b-a814-81bdc2737087-tls-certs\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.383127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.381665 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-home\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.483455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.483415 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jww\" (UniqueName: \"kubernetes.io/projected/fb7b581d-c541-401b-a814-81bdc2737087-kube-api-access-j8jww\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.483671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.483526 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-dshm\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.483671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.483592 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-model-cache\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.483671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.483620 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-tmp-dir\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.483850 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.483705 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7b581d-c541-401b-a814-81bdc2737087-tls-certs\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.483850 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.483828 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-home\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.484086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.484027 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-model-cache\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.484347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.484320 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-home\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.484347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.484342 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-tmp-dir\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.486884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.486853 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7b581d-c541-401b-a814-81bdc2737087-tls-certs\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.487186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.487163 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-dshm\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.492475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.492448 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jww\" (UniqueName: \"kubernetes.io/projected/fb7b581d-c541-401b-a814-81bdc2737087-kube-api-access-j8jww\") pod \"auth-test-model-c8d121-kserve-76d6cd887c-sm8d6\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:39.686119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:39.673104 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:40.051495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:40.051460 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6"] Nov 01 16:19:40.060171 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:19:40.060133 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb7b581d_c541_401b_a814_81bdc2737087.slice/crio-0af90e17bf7682561715f8d478563b507529ce64411e3e0b128816fb032a9c02 WatchSource:0}: Error finding container 0af90e17bf7682561715f8d478563b507529ce64411e3e0b128816fb032a9c02: Status 404 returned error can't find the container with id 0af90e17bf7682561715f8d478563b507529ce64411e3e0b128816fb032a9c02 Nov 01 16:19:40.869696 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:40.869626 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:19:40.882196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:40.882160 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" event={"ID":"fb7b581d-c541-401b-a814-81bdc2737087","Type":"ContainerStarted","Data":"fbcc83c894b04fc0aec917580b7662807010c4f13be5e9a4e9cbc8d9cec471c5"} Nov 01 16:19:40.882377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:40.882213 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" event={"ID":"fb7b581d-c541-401b-a814-81bdc2737087","Type":"ContainerStarted","Data":"0af90e17bf7682561715f8d478563b507529ce64411e3e0b128816fb032a9c02"} Nov 01 16:19:40.923784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:40.923710 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" podStartSLOduration=1.923688255 podStartE2EDuration="1.923688255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:40.902281647 +0000 UTC m=+1921.046736289" watchObservedRunningTime="2026-11-01 16:19:40.923688255 +0000 UTC m=+1921.068142900" Nov 01 16:19:41.612811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:41.612774 4911 scope.go:122] "RemoveContainer" containerID="32c88a35dff80ca9776af9bcfc891e313199d75534c025b78b81198f9bd4891a" Nov 01 16:19:41.676575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:41.676550 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:45.234090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:45.231413 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp"] Nov 01 16:19:45.234090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:45.231705 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" containerID="cri-o://81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b" gracePeriod=30 Nov 01 16:19:49.683470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:49.683413 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:49.696391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:49.696361 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:50.006540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.006507 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:19:50.558793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.558765 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.670806 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-kube-api-access-b7gm4\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-kube-api-access-b7gm4\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671487 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-data\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671595 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istiod-ca-cert\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671642 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-podinfo\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671718 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-token\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671750 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-credential-socket\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671863 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-socket\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671941 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-envoy\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.672058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.671994 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-certs\") pod \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\" (UID: \"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2\") " Nov 01 16:19:50.675109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.673141 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-certs" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.675109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.673346 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-data" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.675109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.673626 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istiod-ca-cert" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:50.676713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.675692 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-credential-socket" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.676713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.676096 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-kube-api-access-b7gm4" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "kube-api-access-b7gm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:50.676713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.676449 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-socket" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.679770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.679741 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-podinfo" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:50.687266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.686642 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-token" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:50.691274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.691227 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-envoy" pod "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" (UID: "a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773826 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7gm4\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-kube-api-access-b7gm4\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773860 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773873 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773882 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773891 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773899 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773907 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.773912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773915 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.774355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:50.773924 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.006518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.006435 4911 generic.go:356] "Generic (PLEG): container finished" podID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerID="81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b" exitCode=0 Nov 01 16:19:51.006518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.006498 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" Nov 01 16:19:51.006753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.006516 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" event={"ID":"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2","Type":"ContainerDied","Data":"81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b"} Nov 01 16:19:51.006753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.006582 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp" event={"ID":"a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2","Type":"ContainerDied","Data":"f875042947ecbd305dbac13163fde81d5aefa03c6ee7029d00e25873a5e07593"} Nov 01 16:19:51.006753 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.006605 4911 scope.go:122] "RemoveContainer" containerID="81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b" Nov 01 16:19:51.022642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.022331 4911 scope.go:122] "RemoveContainer" containerID="2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e" Nov 01 16:19:51.034857 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.034833 4911 scope.go:122] "RemoveContainer" containerID="81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b" Nov 01 16:19:51.035229 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:51.035208 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b\": container with ID starting with 81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b not found: ID does not exist" containerID="81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b" Nov 01 16:19:51.035292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.035246 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b"} err="failed to get container status \"81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b\": rpc error: code = NotFound desc = could not find container \"81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b\": container with ID starting with 81f966012fcb652910110ecb4d3211179807379b54a3289e7dca4596ef7e9f5b not found: ID does not exist" Nov 01 16:19:51.035292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.035266 4911 scope.go:122] "RemoveContainer" containerID="2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e" Nov 01 16:19:51.035528 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:19:51.035515 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e\": container with ID starting with 2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e not found: ID does not exist" containerID="2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e" Nov 01 16:19:51.035585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.035532 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e"} err="failed to get container status \"2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e\": rpc error: code = NotFound desc = could not find container \"2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e\": container with ID starting with 2f0b2bc0c70c28c7eb50dd29e5ced237347288297e2d633ae149e0aa3493b27e not found: ID does not exist" Nov 01 16:19:51.039109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.039086 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp"] Nov 01 16:19:51.043401 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.043377 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-784444-openshift-default-c75d4cbd5-dd5kp"] Nov 01 16:19:51.969353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:19:51.969310 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" path="/var/lib/kubelet/pods/a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2/volumes" Nov 01 16:20:04.153525 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.152885 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l"] Nov 01 16:20:04.153525 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.153157 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" podUID="73011146-c739-4bff-bbea-9a44dd98b493" containerName="maas-api" containerID="cri-o://fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6" gracePeriod=30 Nov 01 16:20:04.722874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.722842 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:20:04.812745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.812708 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-maas-api-tls\") pod \"73011146-c739-4bff-bbea-9a44dd98b493\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " Nov 01 16:20:04.812936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.812905 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73011146-c739-4bff-bbea-9a44dd98b493-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73011146-c739-4bff-bbea-9a44dd98b493-openshift-service-ca\") pod \"73011146-c739-4bff-bbea-9a44dd98b493\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " Nov 01 16:20:04.812991 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.812955 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73011146-c739-4bff-bbea-9a44dd98b493-kube-api-access-phz9b\" (UniqueName: \"kubernetes.io/projected/73011146-c739-4bff-bbea-9a44dd98b493-kube-api-access-phz9b\") pod \"73011146-c739-4bff-bbea-9a44dd98b493\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " Nov 01 16:20:04.813281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.813252 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-metrics-certs\") pod \"73011146-c739-4bff-bbea-9a44dd98b493\" (UID: \"73011146-c739-4bff-bbea-9a44dd98b493\") " Nov 01 16:20:04.813420 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.813287 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73011146-c739-4bff-bbea-9a44dd98b493-openshift-service-ca" pod "73011146-c739-4bff-bbea-9a44dd98b493" (UID: "73011146-c739-4bff-bbea-9a44dd98b493"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:04.813882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.813859 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73011146-c739-4bff-bbea-9a44dd98b493-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:04.815479 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.815452 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-metrics-certs" pod "73011146-c739-4bff-bbea-9a44dd98b493" (UID: "73011146-c739-4bff-bbea-9a44dd98b493"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:04.815598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.815488 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73011146-c739-4bff-bbea-9a44dd98b493-kube-api-access-phz9b" pod "73011146-c739-4bff-bbea-9a44dd98b493" (UID: "73011146-c739-4bff-bbea-9a44dd98b493"). InnerVolumeSpecName "kube-api-access-phz9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:04.815716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.815697 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-maas-api-tls" pod "73011146-c739-4bff-bbea-9a44dd98b493" (UID: "73011146-c739-4bff-bbea-9a44dd98b493"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:04.915706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.915664 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:04.915706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.915707 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73011146-c739-4bff-bbea-9a44dd98b493-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:04.915972 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:04.915724 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phz9b\" (UniqueName: \"kubernetes.io/projected/73011146-c739-4bff-bbea-9a44dd98b493-kube-api-access-phz9b\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:05.141428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.141388 4911 generic.go:356] "Generic (PLEG): container finished" podID="73011146-c739-4bff-bbea-9a44dd98b493" containerID="fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6" exitCode=0 Nov 01 16:20:05.141597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.141448 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" Nov 01 16:20:05.141597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.141463 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" event={"ID":"73011146-c739-4bff-bbea-9a44dd98b493","Type":"ContainerDied","Data":"fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6"} Nov 01 16:20:05.141597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.141540 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l" event={"ID":"73011146-c739-4bff-bbea-9a44dd98b493","Type":"ContainerDied","Data":"699757bf0c43d0ca3e79bfc5397b8c6308a50da81b097823c5bd1bb1cd4169d3"} Nov 01 16:20:05.141597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.141562 4911 scope.go:122] "RemoveContainer" containerID="fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6" Nov 01 16:20:05.156339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.155058 4911 scope.go:122] "RemoveContainer" containerID="fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6" Nov 01 16:20:05.156339 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:20:05.155424 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6\": container with ID starting with fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6 not found: ID does not exist" containerID="fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6" Nov 01 16:20:05.156339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.155460 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6"} err="failed to get container status \"fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6\": rpc error: code = NotFound desc = could not find container \"fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6\": container with ID starting with fbe8ffd3f6fe9f97af22275dfa66be0e29259def2a6552568f7062bbd6384dc6 not found: ID does not exist" Nov 01 16:20:05.165187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.165157 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l"] Nov 01 16:20:05.169309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.169286 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99afca-6659f7d654-4ql5l"] Nov 01 16:20:05.969831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:05.969797 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73011146-c739-4bff-bbea-9a44dd98b493" path="/var/lib/kubelet/pods/73011146-c739-4bff-bbea-9a44dd98b493/volumes" Nov 01 16:20:10.868117 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:20:10.868073 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:20:11.590441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:11.590407 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg"] Nov 01 16:20:11.590694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:11.590663 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" containerID="cri-o://e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994" gracePeriod=30 Nov 01 16:20:13.993010 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:13.992971 4911 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:20:13.993433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:13.993034 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:20:15.048033 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.047996 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc"] Nov 01 16:20:15.048705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048690 4911 cpu_manager.go:414] "containerMap: removing container" podUID="73011146-c739-4bff-bbea-9a44dd98b493" containerName="maas-api" Nov 01 16:20:15.048757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048706 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73011146-c739-4bff-bbea-9a44dd98b493" containerName="maas-api" Nov 01 16:20:15.048757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048731 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" Nov 01 16:20:15.048757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048737 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" Nov 01 16:20:15.048851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048758 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" Nov 01 16:20:15.048851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048763 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" Nov 01 16:20:15.048919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048872 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="73011146-c739-4bff-bbea-9a44dd98b493" containerName="maas-api" Nov 01 16:20:15.048919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048884 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" Nov 01 16:20:15.048919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.048893 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fb0229-2b6f-49cb-b2da-0cf1a786f5b2" containerName="istio-proxy" Nov 01 16:20:15.053953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.053930 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.058026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.057994 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-10b00a-openshift-default-dockercfg-tv6qt\"" Nov 01 16:20:15.062681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.062556 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc"] Nov 01 16:20:15.142896 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.142859 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-credential-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143057 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-kube-api-access-5cccj\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143115 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-certs\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143201 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-envoy\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143230 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f60191-575d-4b42-837b-ddcfe2325251-istiod-ca-cert\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143269 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f60191-575d-4b42-837b-ddcfe2325251-istio-podinfo\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143399 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143433 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-data\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.143465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.143450 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-istio-token\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245319 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245361 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-data\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245379 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-istio-token\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245509 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-credential-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245594 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-kube-api-access-5cccj\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245626 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-certs\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245651 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-envoy\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245820 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-data\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245866 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245861 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f60191-575d-4b42-837b-ddcfe2325251-istiod-ca-cert\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245923 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f60191-575d-4b42-837b-ddcfe2325251-istio-podinfo\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.245938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.245925 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-credential-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.246197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.246017 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-certs\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.246556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.246493 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f60191-575d-4b42-837b-ddcfe2325251-istiod-ca-cert\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.248107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.248080 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-envoy\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.248386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.248368 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f60191-575d-4b42-837b-ddcfe2325251-istio-podinfo\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.253536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.253508 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-istio-token\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.253664 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.253647 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-kube-api-access-5cccj\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.373273 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.373177 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:15.667193 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.667160 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc"] Nov 01 16:20:15.672879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.672839 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:15.673003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.672933 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:15.673003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.672973 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:15.746578 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.746546 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv"] Nov 01 16:20:15.752954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.752929 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.763195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.763169 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv"] Nov 01 16:20:15.764034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.763713 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9e044c-openshift-default-dockercfg-4tjb7\"" Nov 01 16:20:15.855113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855030 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855182 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855215 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855380 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wc88\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-kube-api-access-7wc88\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855478 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855522 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855614 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855748 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855731 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.855842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.855829 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.957666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957566 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.957855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957683 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.957855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957733 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.957855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957763 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.957855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957823 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wc88\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-kube-api-access-7wc88\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.958145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957882 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.958145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957929 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.958145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.957961 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.958145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.958009 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.958145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.958034 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.958435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.958315 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.958729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.958698 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.959387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.959331 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.959927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.959900 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.961588 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.961551 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.961738 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.961719 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.973260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.973231 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wc88\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-kube-api-access-7wc88\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:15.973455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:15.973427 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:16.078439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.078400 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:16.272161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.270627 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" event={"ID":"48f60191-575d-4b42-837b-ddcfe2325251","Type":"ContainerStarted","Data":"cdf24a5a050990cfac6d94ad5e03626d534481cc00da7628445ef3abc6ceac68"} Nov 01 16:20:16.272161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.270697 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" event={"ID":"48f60191-575d-4b42-837b-ddcfe2325251","Type":"ContainerStarted","Data":"835eb4857683163852160d0036c29e2d280f8db6314e91d554e2a6bf71af2ee4"} Nov 01 16:20:16.310122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.310058 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podStartSLOduration=1.310028676 podStartE2EDuration="1.310028676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:16.305426697 +0000 UTC m=+1956.449881351" watchObservedRunningTime="2026-11-01 16:20:16.310028676 +0000 UTC m=+1956.454483317" Nov 01 16:20:16.380883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.380849 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:16.383792 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.383758 4911 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 16:20:16.384075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.384009 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 16:20:16.384321 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:20:16.384301 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307adf41_f7d6_4b51_a0b4_a0bfb1c92728.slice/crio-8b2dbddae426b368821e1ed382f9101d7ab779757186a18982527b222fecfce2 WatchSource:0}: Error finding container 8b2dbddae426b368821e1ed382f9101d7ab779757186a18982527b222fecfce2: Status 404 returned error can't find the container with id 8b2dbddae426b368821e1ed382f9101d7ab779757186a18982527b222fecfce2 Nov 01 16:20:16.386530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.386503 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv"] Nov 01 16:20:16.388342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.388284 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:16.388540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.388516 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:16.388678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:16.388659 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:20:17.047562 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.047510 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:20:17.128386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128109 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-credential-socket\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.128386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128214 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-socket\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.128386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128316 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-data\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.128386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128380 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istiod-ca-cert\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.129817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128425 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-certs\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.129817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128473 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-token\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-token\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.129817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128533 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-podinfo\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.129817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128567 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-kube-api-access-csd7q\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-kube-api-access-csd7q\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.129817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.128617 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-envoy\") pod \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\" (UID: \"19d53d81-84e1-47d0-b62e-29fe3e8ef937\") " Nov 01 16:20:17.129817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.129601 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-data" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.129817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.129762 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-socket" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.130410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.129835 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-certs" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.130410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.129968 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-credential-socket" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.130410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.129524 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istiod-ca-cert" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:17.132486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.132461 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-kube-api-access-csd7q" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "kube-api-access-csd7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:17.132624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.132598 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-podinfo" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:20:17.135469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.135443 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-envoy" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:17.136873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.136844 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-token" pod "19d53d81-84e1-47d0-b62e-29fe3e8ef937" (UID: "19d53d81-84e1-47d0-b62e-29fe3e8ef937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:17.230499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230459 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230499 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230514 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230528 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230587 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230613 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230623 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230634 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csd7q\" (UniqueName: \"kubernetes.io/projected/19d53d81-84e1-47d0-b62e-29fe3e8ef937-kube-api-access-csd7q\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.230717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.230645 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d53d81-84e1-47d0-b62e-29fe3e8ef937-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.374830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.374350 4911 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 16:20:17.374830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.374414 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 16:20:17.416633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.416534 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" event={"ID":"307adf41-f7d6-4b51-a0b4-a0bfb1c92728","Type":"ContainerStarted","Data":"37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe"} Nov 01 16:20:17.416812 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.416647 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" event={"ID":"307adf41-f7d6-4b51-a0b4-a0bfb1c92728","Type":"ContainerStarted","Data":"8b2dbddae426b368821e1ed382f9101d7ab779757186a18982527b222fecfce2"} Nov 01 16:20:17.419347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.419317 4911 generic.go:356] "Generic (PLEG): container finished" podID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerID="e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994" exitCode=0 Nov 01 16:20:17.419510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.419369 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" Nov 01 16:20:17.419510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.419402 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" event={"ID":"19d53d81-84e1-47d0-b62e-29fe3e8ef937","Type":"ContainerDied","Data":"e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994"} Nov 01 16:20:17.419510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.419462 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg" event={"ID":"19d53d81-84e1-47d0-b62e-29fe3e8ef937","Type":"ContainerDied","Data":"46fb2fec24f3e3b65ff60c51df1c09ce3722c13714726423c9fb00a2c883b6ae"} Nov 01 16:20:17.419510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.419493 4911 scope.go:122] "RemoveContainer" containerID="e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994" Nov 01 16:20:17.434557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.434538 4911 scope.go:122] "RemoveContainer" containerID="e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994" Nov 01 16:20:17.434862 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:20:17.434842 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994\": container with ID starting with e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994 not found: ID does not exist" containerID="e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994" Nov 01 16:20:17.434937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.434875 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994"} err="failed to get container status \"e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994\": rpc error: code = NotFound desc = could not find container \"e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994\": container with ID starting with e12f02ff31e2689de975db02a9186a8a9ea0b67f56d457afe032e9ac1a1a4994 not found: ID does not exist" Nov 01 16:20:17.442547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.442506 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podStartSLOduration=2.442493522 podStartE2EDuration="2.442493522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:17.438209713 +0000 UTC m=+1957.582664354" watchObservedRunningTime="2026-11-01 16:20:17.442493522 +0000 UTC m=+1957.586948217" Nov 01 16:20:17.502271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.498478 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg"] Nov 01 16:20:17.505560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:17.505523 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-dw8pg"] Nov 01 16:20:18.007330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:18.007295 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" path="/var/lib/kubelet/pods/19d53d81-84e1-47d0-b62e-29fe3e8ef937/volumes" Nov 01 16:20:18.082062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:18.081889 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:18.082062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:18.082019 4911 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 01 16:20:18.082353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:18.082092 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 01 16:20:18.375514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:18.375415 4911 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 16:20:18.376152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:18.376112 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 16:20:19.085523 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:19.085480 4911 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 01 16:20:19.085934 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:19.085900 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 01 16:20:19.377747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:19.376444 4911 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 16:20:19.377747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:19.376513 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 16:20:20.090062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:20.079003 4911 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 01 16:20:20.090062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:20.079103 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 01 16:20:20.386124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:20.374379 4911 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 16:20:20.386124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:20.374440 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 16:20:21.092067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:21.091941 4911 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 01 16:20:21.092275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:21.092091 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 01 16:20:21.377124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:21.374721 4911 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 16:20:21.377124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:21.374797 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 16:20:22.080704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:22.080666 4911 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 01 16:20:22.081262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:22.080744 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 01 16:20:22.581636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:22.581598 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:22.583943 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:22.583916 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:22.585650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:22.585628 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:20:23.078954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:23.078914 4911 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 01 16:20:23.079139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:23.078994 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 01 16:20:24.084512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:24.084482 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:24.084997 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:24.084975 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:24.085824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:24.085798 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:20:40.903064 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:20:40.902017 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:20:41.957596 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.957558 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp"] Nov 01 16:20:41.959891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.959858 4911 cpu_manager.go:414] "containerMap: removing container" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" Nov 01 16:20:41.959891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.959888 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" Nov 01 16:20:41.960281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.960081 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d53d81-84e1-47d0-b62e-29fe3e8ef937" containerName="istio-proxy" Nov 01 16:20:41.976124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.974174 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp"] Nov 01 16:20:41.976410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.976387 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:41.993528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.993498 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"kube-root-ca.crt\"" Nov 01 16:20:41.995291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.995263 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"default-dockercfg-7tn47\"" Nov 01 16:20:41.995508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:41.995486 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"auth-test-model-a9a967-kserve-self-signed-certs\"" Nov 01 16:20:42.068798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.066927 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-home\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.068798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.067032 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-model-cache\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.068798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.067196 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f4d856e-e307-4e7c-9149-e8d418fc599b-tls-certs\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.068798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.067323 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-tmp-dir\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.068798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.067399 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-dshm\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.068798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.067468 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqk4\" (UniqueName: \"kubernetes.io/projected/1f4d856e-e307-4e7c-9149-e8d418fc599b-kube-api-access-6gqk4\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.087844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.087814 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"openshift-service-ca.crt\"" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.169947 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-tmp-dir\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.170030 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-dshm\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.170115 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqk4\" (UniqueName: \"kubernetes.io/projected/1f4d856e-e307-4e7c-9149-e8d418fc599b-kube-api-access-6gqk4\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.170184 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-home\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.170259 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-model-cache\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.170396 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f4d856e-e307-4e7c-9149-e8d418fc599b-tls-certs\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.173470 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-home\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.173724 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-model-cache\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.174108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.173977 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-tmp-dir\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.201060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.200103 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-dshm\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.201060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.200885 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f4d856e-e307-4e7c-9149-e8d418fc599b-tls-certs\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.216208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.209886 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqk4\" (UniqueName: \"kubernetes.io/projected/1f4d856e-e307-4e7c-9149-e8d418fc599b-kube-api-access-6gqk4\") pod \"auth-test-model-a9a967-kserve-7c978cdd97-qwbpp\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.324183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.321453 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:42.727142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:42.727106 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp"] Nov 01 16:20:42.727603 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:20:42.727575 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f4d856e_e307_4e7c_9149_e8d418fc599b.slice/crio-6ce9d4f99d8a852394de626c954eef22f9795d3e40ed9ee175b66c072ada8aeb WatchSource:0}: Error finding container 6ce9d4f99d8a852394de626c954eef22f9795d3e40ed9ee175b66c072ada8aeb: Status 404 returned error can't find the container with id 6ce9d4f99d8a852394de626c954eef22f9795d3e40ed9ee175b66c072ada8aeb Nov 01 16:20:43.504064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:43.497840 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v"] Nov 01 16:20:43.767330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:43.766927 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" event={"ID":"1f4d856e-e307-4e7c-9149-e8d418fc599b","Type":"ContainerStarted","Data":"f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29"} Nov 01 16:20:43.767330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:43.766984 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" event={"ID":"1f4d856e-e307-4e7c-9149-e8d418fc599b","Type":"ContainerStarted","Data":"6ce9d4f99d8a852394de626c954eef22f9795d3e40ed9ee175b66c072ada8aeb"} Nov 01 16:20:43.814817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:43.814763 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" podStartSLOduration=2.814746589 podStartE2EDuration="2.814746589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:43.811637218 +0000 UTC m=+1983.956091869" watchObservedRunningTime="2026-11-01 16:20:43.814746589 +0000 UTC m=+1983.959201232" Nov 01 16:20:44.321719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:44.321681 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:52.322506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:52.322460 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:52.335392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:52.335362 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:52.852811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:52.852779 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:20:58.539404 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.539364 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" podUID="2c2e75c6-d218-4d2c-a851-32ee17c455df" containerName="main" containerID="cri-o://7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55" gracePeriod=45 Nov 01 16:20:58.828152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.828126 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:20:58.930007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.929970 4911 generic.go:356] "Generic (PLEG): container finished" podID="2c2e75c6-d218-4d2c-a851-32ee17c455df" containerID="7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55" exitCode=0 Nov 01 16:20:58.930234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.930022 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" Nov 01 16:20:58.930234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.930076 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" event={"ID":"2c2e75c6-d218-4d2c-a851-32ee17c455df","Type":"ContainerDied","Data":"7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55"} Nov 01 16:20:58.930234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.930139 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v" event={"ID":"2c2e75c6-d218-4d2c-a851-32ee17c455df","Type":"ContainerDied","Data":"cf10e13c8e60417337651fcd956b61c5e036972add3c16b5914d491a7040e01c"} Nov 01 16:20:58.930234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.930165 4911 scope.go:122] "RemoveContainer" containerID="7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55" Nov 01 16:20:58.941033 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.941012 4911 scope.go:122] "RemoveContainer" containerID="7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55" Nov 01 16:20:58.941336 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:20:58.941311 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55\": container with ID starting with 7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55 not found: ID does not exist" containerID="7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55" Nov 01 16:20:58.941395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.941345 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55"} err="failed to get container status \"7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55\": rpc error: code = NotFound desc = could not find container \"7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55\": container with ID starting with 7ade8eb676931e3ad9a6e4425a9ffbb2907f4971a66d3cc2cd1cdcd29f4c3d55 not found: ID does not exist" Nov 01 16:20:58.978707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.978677 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-dshm\") pod \"2c2e75c6-d218-4d2c-a851-32ee17c455df\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " Nov 01 16:20:58.978932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.978910 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-model-cache\") pod \"2c2e75c6-d218-4d2c-a851-32ee17c455df\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " Nov 01 16:20:58.979092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979070 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2e75c6-d218-4d2c-a851-32ee17c455df-kube-api-access-h8fr9\" (UniqueName: \"kubernetes.io/projected/2c2e75c6-d218-4d2c-a851-32ee17c455df-kube-api-access-h8fr9\") pod \"2c2e75c6-d218-4d2c-a851-32ee17c455df\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " Nov 01 16:20:58.979163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979148 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-tmp-dir\") pod \"2c2e75c6-d218-4d2c-a851-32ee17c455df\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " Nov 01 16:20:58.979214 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979195 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-model-cache" pod "2c2e75c6-d218-4d2c-a851-32ee17c455df" (UID: "2c2e75c6-d218-4d2c-a851-32ee17c455df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:58.979274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979202 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-home\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-home\") pod \"2c2e75c6-d218-4d2c-a851-32ee17c455df\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " Nov 01 16:20:58.979332 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979277 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c2e75c6-d218-4d2c-a851-32ee17c455df-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2e75c6-d218-4d2c-a851-32ee17c455df-tls-certs\") pod \"2c2e75c6-d218-4d2c-a851-32ee17c455df\" (UID: \"2c2e75c6-d218-4d2c-a851-32ee17c455df\") " Nov 01 16:20:58.979391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979349 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-home" pod "2c2e75c6-d218-4d2c-a851-32ee17c455df" (UID: "2c2e75c6-d218-4d2c-a851-32ee17c455df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:58.979453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979427 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-tmp-dir" pod "2c2e75c6-d218-4d2c-a851-32ee17c455df" (UID: "2c2e75c6-d218-4d2c-a851-32ee17c455df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:58.979966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979943 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:58.979966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979965 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:58.980224 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.979975 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:58.981379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.981355 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-dshm" pod "2c2e75c6-d218-4d2c-a851-32ee17c455df" (UID: "2c2e75c6-d218-4d2c-a851-32ee17c455df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:58.981494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.981424 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2e75c6-d218-4d2c-a851-32ee17c455df-kube-api-access-h8fr9" pod "2c2e75c6-d218-4d2c-a851-32ee17c455df" (UID: "2c2e75c6-d218-4d2c-a851-32ee17c455df"). InnerVolumeSpecName "kube-api-access-h8fr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:58.981672 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:58.981653 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2e75c6-d218-4d2c-a851-32ee17c455df-tls-certs" pod "2c2e75c6-d218-4d2c-a851-32ee17c455df" (UID: "2c2e75c6-d218-4d2c-a851-32ee17c455df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:59.092488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:59.091773 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8fr9\" (UniqueName: \"kubernetes.io/projected/2c2e75c6-d218-4d2c-a851-32ee17c455df-kube-api-access-h8fr9\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:59.092488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:59.091815 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2e75c6-d218-4d2c-a851-32ee17c455df-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:59.092488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:59.091830 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2e75c6-d218-4d2c-a851-32ee17c455df-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:59.278060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:59.259166 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v"] Nov 01 16:20:59.278060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:59.266183 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-10b00a/test-model-10b00a-kserve-57f6c46867-25n7v"] Nov 01 16:20:59.970294 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:20:59.970258 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2e75c6-d218-4d2c-a851-32ee17c455df" path="/var/lib/kubelet/pods/2c2e75c6-d218-4d2c-a851-32ee17c455df/volumes" Nov 01 16:21:10.879169 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:21:10.879131 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:21:25.577442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:25.577350 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd"] Nov 01 16:21:38.226729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:38.226681 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc"] Nov 01 16:21:38.227290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:38.226953 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" containerID="cri-o://cdf24a5a050990cfac6d94ad5e03626d534481cc00da7628445ef3abc6ceac68" gracePeriod=30 Nov 01 16:21:40.601124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.601026 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" podUID="97b70d41-ad60-43ee-8396-b2c7610555d9" containerName="main" containerID="cri-o://dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da" gracePeriod=45 Nov 01 16:21:40.659062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.658765 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" podUID="97b70d41-ad60-43ee-8396-b2c7610555d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": dial tcp 10.132.0.163:8000: connect: connection refused" Nov 01 16:21:40.959805 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:21:40.959719 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:21:40.970913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.970871 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:21:40.999051 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999002 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b70d41-ad60-43ee-8396-b2c7610555d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/97b70d41-ad60-43ee-8396-b2c7610555d9-tls-certs\") pod \"97b70d41-ad60-43ee-8396-b2c7610555d9\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " Nov 01 16:21:40.999233 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999070 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-model-cache\") pod \"97b70d41-ad60-43ee-8396-b2c7610555d9\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " Nov 01 16:21:40.999329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999309 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-model-cache" pod "97b70d41-ad60-43ee-8396-b2c7610555d9" (UID: "97b70d41-ad60-43ee-8396-b2c7610555d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:40.999428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999411 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-tmp-dir\") pod \"97b70d41-ad60-43ee-8396-b2c7610555d9\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " Nov 01 16:21:40.999505 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999488 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b70d41-ad60-43ee-8396-b2c7610555d9-kube-api-access-wx2cg\" (UniqueName: \"kubernetes.io/projected/97b70d41-ad60-43ee-8396-b2c7610555d9-kube-api-access-wx2cg\") pod \"97b70d41-ad60-43ee-8396-b2c7610555d9\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " Nov 01 16:21:40.999703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999685 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-dshm\") pod \"97b70d41-ad60-43ee-8396-b2c7610555d9\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " Nov 01 16:21:40.999829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999813 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-home\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-home\") pod \"97b70d41-ad60-43ee-8396-b2c7610555d9\" (UID: \"97b70d41-ad60-43ee-8396-b2c7610555d9\") " Nov 01 16:21:41.000107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:40.999685 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-tmp-dir" pod "97b70d41-ad60-43ee-8396-b2c7610555d9" (UID: "97b70d41-ad60-43ee-8396-b2c7610555d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:41.000308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.000286 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-home" pod "97b70d41-ad60-43ee-8396-b2c7610555d9" (UID: "97b70d41-ad60-43ee-8396-b2c7610555d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:41.000590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.000573 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:41.000679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.000594 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:41.000679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.000604 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:41.001955 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.001927 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b70d41-ad60-43ee-8396-b2c7610555d9-tls-certs" pod "97b70d41-ad60-43ee-8396-b2c7610555d9" (UID: "97b70d41-ad60-43ee-8396-b2c7610555d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:41.002609 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.002582 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b70d41-ad60-43ee-8396-b2c7610555d9-kube-api-access-wx2cg" pod "97b70d41-ad60-43ee-8396-b2c7610555d9" (UID: "97b70d41-ad60-43ee-8396-b2c7610555d9"). InnerVolumeSpecName "kube-api-access-wx2cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:41.002770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.002597 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-dshm" pod "97b70d41-ad60-43ee-8396-b2c7610555d9" (UID: "97b70d41-ad60-43ee-8396-b2c7610555d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:41.102624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.102585 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2cg\" (UniqueName: \"kubernetes.io/projected/97b70d41-ad60-43ee-8396-b2c7610555d9-kube-api-access-wx2cg\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:41.102624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.102624 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b70d41-ad60-43ee-8396-b2c7610555d9-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:41.102831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.102639 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b70d41-ad60-43ee-8396-b2c7610555d9-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:41.525325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.525288 4911 generic.go:356] "Generic (PLEG): container finished" podID="97b70d41-ad60-43ee-8396-b2c7610555d9" containerID="dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da" exitCode=0 Nov 01 16:21:41.525546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.525340 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" Nov 01 16:21:41.525546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.525374 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" event={"ID":"97b70d41-ad60-43ee-8396-b2c7610555d9","Type":"ContainerDied","Data":"dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da"} Nov 01 16:21:41.525546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.525419 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd" event={"ID":"97b70d41-ad60-43ee-8396-b2c7610555d9","Type":"ContainerDied","Data":"27149370bb47bf647ad71b98ae7352a7a383561dfa2da96518d5fe96f593073a"} Nov 01 16:21:41.525546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.525437 4911 scope.go:122] "RemoveContainer" containerID="dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da" Nov 01 16:21:41.543659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.543633 4911 scope.go:122] "RemoveContainer" containerID="dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da" Nov 01 16:21:41.544110 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:21:41.544075 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da\": container with ID starting with dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da not found: ID does not exist" containerID="dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da" Nov 01 16:21:41.544229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.544116 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da"} err="failed to get container status \"dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da\": rpc error: code = NotFound desc = could not find container \"dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da\": container with ID starting with dbb5844fb922815e3e04c13afd7f71d38c6c6e1f34d47c8432ad39dc1b1620da not found: ID does not exist" Nov 01 16:21:41.556778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.556748 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd"] Nov 01 16:21:41.560992 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.560963 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a4240a/test-model-a4240a-kserve-5585b759ff-657vd"] Nov 01 16:21:41.969675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:41.969590 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b70d41-ad60-43ee-8396-b2c7610555d9" path="/var/lib/kubelet/pods/97b70d41-ad60-43ee-8396-b2c7610555d9/volumes" Nov 01 16:21:42.653062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:42.652511 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4"] Nov 01 16:21:43.551336 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.551252 4911 generic.go:356] "Generic (PLEG): container finished" podID="48f60191-575d-4b42-837b-ddcfe2325251" containerID="cdf24a5a050990cfac6d94ad5e03626d534481cc00da7628445ef3abc6ceac68" exitCode=0 Nov 01 16:21:43.551797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.551395 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" event={"ID":"48f60191-575d-4b42-837b-ddcfe2325251","Type":"ContainerDied","Data":"cdf24a5a050990cfac6d94ad5e03626d534481cc00da7628445ef3abc6ceac68"} Nov 01 16:21:43.628488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.628451 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:21:43.771869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.771821 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48f60191-575d-4b42-837b-ddcfe2325251-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f60191-575d-4b42-837b-ddcfe2325251-istio-podinfo\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.771943 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-certs\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772096 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-kube-api-access-5cccj\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772129 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-envoy\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772165 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f60191-575d-4b42-837b-ddcfe2325251-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f60191-575d-4b42-837b-ddcfe2325251-istiod-ca-cert\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772239 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-credential-socket\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772408 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-socket\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772447 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-istio-token\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-istio-token\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.772551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772510 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-data\") pod \"48f60191-575d-4b42-837b-ddcfe2325251\" (UID: \"48f60191-575d-4b42-837b-ddcfe2325251\") " Nov 01 16:21:43.773002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.772965 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-socket" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.773167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.773058 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-credential-socket" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.773236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.773182 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-data" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.773549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.773524 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.773645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.773556 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.773645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.773575 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.773754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.773710 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f60191-575d-4b42-837b-ddcfe2325251-istiod-ca-cert" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:43.774084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.774030 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-certs" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.776006 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.775912 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-kube-api-access-5cccj" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "kube-api-access-5cccj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:43.776006 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.775961 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48f60191-575d-4b42-837b-ddcfe2325251-istio-podinfo" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:21:43.776006 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.775961 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-istio-token" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:43.776356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.776328 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-envoy" pod "48f60191-575d-4b42-837b-ddcfe2325251" (UID: "48f60191-575d-4b42-837b-ddcfe2325251"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.875267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.875175 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f60191-575d-4b42-837b-ddcfe2325251-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.875267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.875213 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.875267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.875230 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f60191-575d-4b42-837b-ddcfe2325251-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.875267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.875244 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.875267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.875258 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/48f60191-575d-4b42-837b-ddcfe2325251-kube-api-access-5cccj\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.875673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:43.875274 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f60191-575d-4b42-837b-ddcfe2325251-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:44.561765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:44.561714 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" event={"ID":"48f60191-575d-4b42-837b-ddcfe2325251","Type":"ContainerDied","Data":"835eb4857683163852160d0036c29e2d280f8db6314e91d554e2a6bf71af2ee4"} Nov 01 16:21:44.562290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:44.561792 4911 scope.go:122] "RemoveContainer" containerID="cdf24a5a050990cfac6d94ad5e03626d534481cc00da7628445ef3abc6ceac68" Nov 01 16:21:44.562290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:44.561728 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc" Nov 01 16:21:44.591106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:44.591073 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc"] Nov 01 16:21:44.600731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:44.600701 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-bvsrc"] Nov 01 16:21:45.262324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.262271 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt"] Nov 01 16:21:45.263370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263343 4911 cpu_manager.go:414] "containerMap: removing container" podUID="2c2e75c6-d218-4d2c-a851-32ee17c455df" containerName="main" Nov 01 16:21:45.263370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263371 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2e75c6-d218-4d2c-a851-32ee17c455df" containerName="main" Nov 01 16:21:45.263597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263393 4911 cpu_manager.go:414] "containerMap: removing container" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" Nov 01 16:21:45.263597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263401 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" Nov 01 16:21:45.263597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263437 4911 cpu_manager.go:414] "containerMap: removing container" podUID="97b70d41-ad60-43ee-8396-b2c7610555d9" containerName="main" Nov 01 16:21:45.263597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263444 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b70d41-ad60-43ee-8396-b2c7610555d9" containerName="main" Nov 01 16:21:45.263750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263658 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2e75c6-d218-4d2c-a851-32ee17c455df" containerName="main" Nov 01 16:21:45.263750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263673 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f60191-575d-4b42-837b-ddcfe2325251" containerName="istio-proxy" Nov 01 16:21:45.263750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.263688 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b70d41-ad60-43ee-8396-b2c7610555d9" containerName="main" Nov 01 16:21:45.269393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.269363 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.276486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.276457 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c78897\"/\"openshift-service-ca.crt\"" Nov 01 16:21:45.277458 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.277411 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt"] Nov 01 16:21:45.280527 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.280458 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c78897\"/\"auth-test-model-c78897-kserve-self-signed-certs\"" Nov 01 16:21:45.280608 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.280541 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c78897\"/\"default-dockercfg-xknr6\"" Nov 01 16:21:45.376478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.376443 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c78897\"/\"kube-root-ca.crt\"" Nov 01 16:21:45.410097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.410056 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-dshm\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.410277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.410186 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tql\" (UniqueName: \"kubernetes.io/projected/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-kube-api-access-p2tql\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.410360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.410312 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-model-cache\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.410360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.410353 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-home\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.410450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.410395 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tmp-dir\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.410573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.410553 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tls-certs\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.516215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.516111 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tls-certs\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.516559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.516531 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-dshm\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.516743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.516725 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tql\" (UniqueName: \"kubernetes.io/projected/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-kube-api-access-p2tql\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.516985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.516956 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-model-cache\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.517177 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.517001 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-home\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.517177 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.517062 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tmp-dir\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.517637 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.517611 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tmp-dir\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.518300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.518279 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-model-cache\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.518697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.518677 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-home\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.527713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.527685 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tls-certs\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.533309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.533282 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-dshm\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.549677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.549642 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tql\" (UniqueName: \"kubernetes.io/projected/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-kube-api-access-p2tql\") pod \"auth-test-model-c78897-kserve-58d59fc4d5-9q9xt\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.591723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.591674 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:45.890279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.890202 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt"] Nov 01 16:21:45.890628 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:21:45.890585 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c9c93d6_86b7_4cf7_b73f_46e5737dd48e.slice/crio-a456fec77c980991ed67d0e056acf7fa6b94f4d15820141efb83554b324d6781 WatchSource:0}: Error finding container a456fec77c980991ed67d0e056acf7fa6b94f4d15820141efb83554b324d6781: Status 404 returned error can't find the container with id a456fec77c980991ed67d0e056acf7fa6b94f4d15820141efb83554b324d6781 Nov 01 16:21:45.972087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:45.972055 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f60191-575d-4b42-837b-ddcfe2325251" path="/var/lib/kubelet/pods/48f60191-575d-4b42-837b-ddcfe2325251/volumes" Nov 01 16:21:46.594352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:46.594309 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" event={"ID":"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e","Type":"ContainerStarted","Data":"0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648"} Nov 01 16:21:46.594352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:46.594354 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" event={"ID":"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e","Type":"ContainerStarted","Data":"a456fec77c980991ed67d0e056acf7fa6b94f4d15820141efb83554b324d6781"} Nov 01 16:21:46.612066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:46.611999 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" podStartSLOduration=1.611984812 podStartE2EDuration="1.611984812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:46.610226542 +0000 UTC m=+2046.754681184" watchObservedRunningTime="2026-11-01 16:21:46.611984812 +0000 UTC m=+2046.756439475" Nov 01 16:21:47.448305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.448269 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr"] Nov 01 16:21:47.453454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.453430 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.470226 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.470197 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c78897-openshift-default-dockercfg-q2vpn\"" Nov 01 16:21:47.474391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.474358 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr"] Nov 01 16:21:47.552880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.552841 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.552902 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0105a45c-db4b-4929-950e-4714d7ac9369-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.553100 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhd2x\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-kube-api-access-mhd2x\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.553150 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.553226 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.553279 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.553304 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.553394 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.553474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.553430 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0105a45c-db4b-4929-950e-4714d7ac9369-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.592613 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.592582 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:47.655201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655155 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655232 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0105a45c-db4b-4929-950e-4714d7ac9369-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655344 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhd2x\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-kube-api-access-mhd2x\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655396 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655428 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655504 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655550 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655624 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.655679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.655670 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0105a45c-db4b-4929-950e-4714d7ac9369-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.656221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.656023 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.656221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.656107 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.656221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.656143 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0105a45c-db4b-4929-950e-4714d7ac9369-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.656221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.656179 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.656414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.656335 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.657674 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.657650 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.658254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.658235 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0105a45c-db4b-4929-950e-4714d7ac9369-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.664929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.664904 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhd2x\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-kube-api-access-mhd2x\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.668091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.668024 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:47.795607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:47.795572 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:48.088115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.088085 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr"] Nov 01 16:21:48.088625 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:21:48.088600 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0105a45c_db4b_4929_950e_4714d7ac9369.slice/crio-c44d34e59d96cfe7398c42583bc5c6b4b13f62d7852abaed9c62359fb72f6528 WatchSource:0}: Error finding container c44d34e59d96cfe7398c42583bc5c6b4b13f62d7852abaed9c62359fb72f6528: Status 404 returned error can't find the container with id c44d34e59d96cfe7398c42583bc5c6b4b13f62d7852abaed9c62359fb72f6528 Nov 01 16:21:48.092102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.092064 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.092216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.092146 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.092216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.092195 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.613933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.613883 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" event={"ID":"0105a45c-db4b-4929-950e-4714d7ac9369","Type":"ContainerStarted","Data":"73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68"} Nov 01 16:21:48.614148 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.613954 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" event={"ID":"0105a45c-db4b-4929-950e-4714d7ac9369","Type":"ContainerStarted","Data":"c44d34e59d96cfe7398c42583bc5c6b4b13f62d7852abaed9c62359fb72f6528"} Nov 01 16:21:48.641433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.641370 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" podStartSLOduration=1.641350225 podStartE2EDuration="1.641350225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:48.635056371 +0000 UTC m=+2048.779511015" watchObservedRunningTime="2026-11-01 16:21:48.641350225 +0000 UTC m=+2048.785804868" Nov 01 16:21:48.796141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:48.796102 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:49.797464 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:49.797430 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 01 16:21:49.797886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:49.797484 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 01 16:21:50.796706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:50.796665 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 01 16:21:50.796886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:50.796734 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 01 16:21:51.796531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:51.796495 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 01 16:21:51.796921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:51.796562 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 01 16:21:52.796288 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:52.796252 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 01 16:21:52.796497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:52.796323 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 01 16:21:53.796305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:53.796267 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 01 16:21:53.796722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:53.796334 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 01 16:21:54.801553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:54.801510 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:54.802054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:54.802023 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:54.802914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:54.802892 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:21:55.592786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:55.592746 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:55.606211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:55.606178 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:55.691221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:55.691189 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:21:57.710196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:57.710155 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" podUID="2908c72c-772e-4003-a90b-7ed121e92979" containerName="main" containerID="cri-o://980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39" gracePeriod=45 Nov 01 16:21:57.867829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:57.867789 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" podUID="2908c72c-772e-4003-a90b-7ed121e92979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.164:8000/ready\": dial tcp 10.132.0.164:8000: connect: connection refused" Nov 01 16:21:58.020922 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.020892 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:21:58.072558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.072520 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2908c72c-772e-4003-a90b-7ed121e92979-tls-certs\" (UniqueName: \"kubernetes.io/secret/2908c72c-772e-4003-a90b-7ed121e92979-tls-certs\") pod \"2908c72c-772e-4003-a90b-7ed121e92979\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " Nov 01 16:21:58.072759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.072619 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-tmp-dir\") pod \"2908c72c-772e-4003-a90b-7ed121e92979\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " Nov 01 16:21:58.072923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.072900 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-tmp-dir" pod "2908c72c-772e-4003-a90b-7ed121e92979" (UID: "2908c72c-772e-4003-a90b-7ed121e92979"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:58.073001 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.072983 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-dshm\") pod \"2908c72c-772e-4003-a90b-7ed121e92979\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " Nov 01 16:21:58.073550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.073019 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2908c72c-772e-4003-a90b-7ed121e92979-kube-api-access-jl2pl\" (UniqueName: \"kubernetes.io/projected/2908c72c-772e-4003-a90b-7ed121e92979-kube-api-access-jl2pl\") pod \"2908c72c-772e-4003-a90b-7ed121e92979\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " Nov 01 16:21:58.073772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.073753 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-model-cache\") pod \"2908c72c-772e-4003-a90b-7ed121e92979\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " Nov 01 16:21:58.075379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.073886 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-home\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-home\") pod \"2908c72c-772e-4003-a90b-7ed121e92979\" (UID: \"2908c72c-772e-4003-a90b-7ed121e92979\") " Nov 01 16:21:58.075379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.074141 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-model-cache" pod "2908c72c-772e-4003-a90b-7ed121e92979" (UID: "2908c72c-772e-4003-a90b-7ed121e92979"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:58.075379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.074812 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:58.075379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.074833 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:58.075599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.075523 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2908c72c-772e-4003-a90b-7ed121e92979-tls-certs" pod "2908c72c-772e-4003-a90b-7ed121e92979" (UID: "2908c72c-772e-4003-a90b-7ed121e92979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:58.075711 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.075685 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-home" pod "2908c72c-772e-4003-a90b-7ed121e92979" (UID: "2908c72c-772e-4003-a90b-7ed121e92979"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:58.077516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.077484 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2908c72c-772e-4003-a90b-7ed121e92979-kube-api-access-jl2pl" pod "2908c72c-772e-4003-a90b-7ed121e92979" (UID: "2908c72c-772e-4003-a90b-7ed121e92979"). InnerVolumeSpecName "kube-api-access-jl2pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:58.077914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.077888 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-dshm" pod "2908c72c-772e-4003-a90b-7ed121e92979" (UID: "2908c72c-772e-4003-a90b-7ed121e92979"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:58.176088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.176007 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2908c72c-772e-4003-a90b-7ed121e92979-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:58.176088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.176058 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:58.176088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.176077 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2pl\" (UniqueName: \"kubernetes.io/projected/2908c72c-772e-4003-a90b-7ed121e92979-kube-api-access-jl2pl\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:58.176088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.176090 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2908c72c-772e-4003-a90b-7ed121e92979-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:58.350593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.350492 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt"] Nov 01 16:21:58.356937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.352537 4911 cpu_manager.go:414] "containerMap: removing container" podUID="2908c72c-772e-4003-a90b-7ed121e92979" containerName="main" Nov 01 16:21:58.356937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.352566 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2908c72c-772e-4003-a90b-7ed121e92979" containerName="main" Nov 01 16:21:58.356937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.352909 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="2908c72c-772e-4003-a90b-7ed121e92979" containerName="main" Nov 01 16:21:58.365472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.365441 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.368225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.365345 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt"] Nov 01 16:21:58.379377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.379349 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"openshift-service-ca.crt\"" Nov 01 16:21:58.379606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.379560 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"sub-test-model-f01de5-kserve-self-signed-certs\"" Nov 01 16:21:58.379716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.379699 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"kube-root-ca.crt\"" Nov 01 16:21:58.380916 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.380889 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"default-dockercfg-mjz9s\"" Nov 01 16:21:58.482077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.482017 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tls-certs\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.482287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.482269 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdvj\" (UniqueName: \"kubernetes.io/projected/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-kube-api-access-qtdvj\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.482363 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.482312 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-model-cache\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.482425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.482377 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-home\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.482425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.482416 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-dshm\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.482553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.482443 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tmp-dir\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.587062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.586462 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdvj\" (UniqueName: \"kubernetes.io/projected/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-kube-api-access-qtdvj\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.587062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.586544 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-model-cache\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.587062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.586607 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-home\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.587062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.586669 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-dshm\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.587062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.586704 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tmp-dir\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.587062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.586772 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tls-certs\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.592184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.592123 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tmp-dir\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.604111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.597431 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-model-cache\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.604111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.598338 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-home\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.604111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.598665 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tls-certs\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.604111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.603257 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-dshm\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.604111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.603613 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdvj\" (UniqueName: \"kubernetes.io/projected/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-kube-api-access-qtdvj\") pod \"sub-test-model-f01de5-kserve-6b4d5fc767-9hljt\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.690125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.687561 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:21:58.709777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.709738 4911 generic.go:356] "Generic (PLEG): container finished" podID="2908c72c-772e-4003-a90b-7ed121e92979" containerID="980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39" exitCode=0 Nov 01 16:21:58.710297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.710268 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" event={"ID":"2908c72c-772e-4003-a90b-7ed121e92979","Type":"ContainerDied","Data":"980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39"} Nov 01 16:21:58.710802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.710775 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" event={"ID":"2908c72c-772e-4003-a90b-7ed121e92979","Type":"ContainerDied","Data":"82f7c662babd204a55ea5b46988cfb920114ae22251dd91caa3272af49336bd4"} Nov 01 16:21:58.710921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.710908 4911 scope.go:122] "RemoveContainer" containerID="980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39" Nov 01 16:21:58.711242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.711223 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4" Nov 01 16:21:58.778094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.778057 4911 scope.go:122] "RemoveContainer" containerID="980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39" Nov 01 16:21:58.778456 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:21:58.778430 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39\": container with ID starting with 980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39 not found: ID does not exist" containerID="980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39" Nov 01 16:21:58.778538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.778468 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39"} err="failed to get container status \"980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39\": rpc error: code = NotFound desc = could not find container \"980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39\": container with ID starting with 980f09d04ca1f18090a2551e4a27f4489df21d8ae9299671a6745300d945fd39 not found: ID does not exist" Nov 01 16:21:58.794515 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.793881 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4"] Nov 01 16:21:58.804243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:58.801265 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0c091d/test-model-0c091d-kserve-5c547c9997-cq9n4"] Nov 01 16:21:59.022477 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:21:59.022438 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1edffe25_b7c8_4f4e_b7fd_84ac4879cfe0.slice/crio-5766c0a7cdb0bdf7b45e5ade2601f9a3c6b2149c0c93e653ed9e96c76e3cc72d WatchSource:0}: Error finding container 5766c0a7cdb0bdf7b45e5ade2601f9a3c6b2149c0c93e653ed9e96c76e3cc72d: Status 404 returned error can't find the container with id 5766c0a7cdb0bdf7b45e5ade2601f9a3c6b2149c0c93e653ed9e96c76e3cc72d Nov 01 16:21:59.022742 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:59.022696 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt"] Nov 01 16:21:59.720803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:59.720762 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" event={"ID":"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0","Type":"ContainerStarted","Data":"0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f"} Nov 01 16:21:59.721305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:59.720818 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" event={"ID":"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0","Type":"ContainerStarted","Data":"5766c0a7cdb0bdf7b45e5ade2601f9a3c6b2149c0c93e653ed9e96c76e3cc72d"} Nov 01 16:21:59.744119 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:59.744066 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" podStartSLOduration=1.7440340650000001 podStartE2EDuration="1.744034065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:59.739425031 +0000 UTC m=+2059.883879698" watchObservedRunningTime="2026-11-01 16:21:59.744034065 +0000 UTC m=+2059.888488705" Nov 01 16:21:59.971227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:21:59.971125 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2908c72c-772e-4003-a90b-7ed121e92979" path="/var/lib/kubelet/pods/2908c72c-772e-4003-a90b-7ed121e92979/volumes" Nov 01 16:22:00.687874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:00.687835 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:22:01.000696 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:01.000660 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2"] Nov 01 16:22:01.001202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:01.000911 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" containerID="cri-o://3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2" gracePeriod=30 Nov 01 16:22:01.164475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:01.164440 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2"] Nov 01 16:22:01.164683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:01.164643 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" containerID="cri-o://1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34" gracePeriod=30 Nov 01 16:22:06.313352 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.313320 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:22:06.412828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.412796 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-socket\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.412876 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/869649a0-d0cd-4236-ab36-567fc7890712-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869649a0-d0cd-4236-ab36-567fc7890712-istio-podinfo\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.412970 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869649a0-d0cd-4236-ab36-567fc7890712-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869649a0-d0cd-4236-ab36-567fc7890712-istiod-ca-cert\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413052 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-data\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413099 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-envoy\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413128 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-certs\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413161 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-istio-token\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-istio-token\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413308 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-kube-api-access-mc6wv\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-kube-api-access-mc6wv\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413123 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-socket" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.413424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413361 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-credential-socket\") pod \"869649a0-d0cd-4236-ab36-567fc7890712\" (UID: \"869649a0-d0cd-4236-ab36-567fc7890712\") " Nov 01 16:22:06.413695 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413332 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-data" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.413800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413436 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869649a0-d0cd-4236-ab36-567fc7890712-istiod-ca-cert" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:06.413880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.413531 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-certs" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.414085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.414023 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-credential-socket" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.414241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.414220 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.414345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.414243 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869649a0-d0cd-4236-ab36-567fc7890712-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.414345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.414254 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.414345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.414263 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.414345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.414271 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.426131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.417470 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/869649a0-d0cd-4236-ab36-567fc7890712-istio-podinfo" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:22:06.426131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.418258 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-envoy" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.426131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.418720 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-istio-token" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.426131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.421465 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-kube-api-access-mc6wv" pod "869649a0-d0cd-4236-ab36-567fc7890712" (UID: "869649a0-d0cd-4236-ab36-567fc7890712"). InnerVolumeSpecName "kube-api-access-mc6wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.509275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.509247 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:22:06.516035 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.516010 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869649a0-d0cd-4236-ab36-567fc7890712-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.516155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.516069 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869649a0-d0cd-4236-ab36-567fc7890712-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.516155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.516082 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.516155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.516092 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc6wv\" (UniqueName: \"kubernetes.io/projected/869649a0-d0cd-4236-ab36-567fc7890712-kube-api-access-mc6wv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.616961 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.616873 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istiod-ca-cert\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.616961 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-credential-socket\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.616992 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-podinfo\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617034 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-envoy\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617115 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-token\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-token\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617152 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-certs\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617190 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-kube-api-access-zzhmc\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-kube-api-access-zzhmc\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617289 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-socket\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617350 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istiod-ca-cert" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:06.617447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617363 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-credential-socket" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.617447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617415 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-data\") pod \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\" (UID: \"7150a085-b0d4-46a0-8a41-cc510b4ba44e\") " Nov 01 16:22:06.617778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617656 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-socket" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.617848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617797 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-certs" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.617848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.617806 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-data" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.618222 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.618198 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.618222 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.618220 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.618391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.618230 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.618391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.618239 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.618391 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.618248 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.619438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.619414 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-kube-api-access-zzhmc" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "kube-api-access-zzhmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.619542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.619507 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-envoy" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.619819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.619793 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-podinfo" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:22:06.619819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.619800 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-token" pod "7150a085-b0d4-46a0-8a41-cc510b4ba44e" (UID: "7150a085-b0d4-46a0-8a41-cc510b4ba44e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.720212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.720173 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.720212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.720204 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.720212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.720215 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhmc\" (UniqueName: \"kubernetes.io/projected/7150a085-b0d4-46a0-8a41-cc510b4ba44e-kube-api-access-zzhmc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.720446 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.720226 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7150a085-b0d4-46a0-8a41-cc510b4ba44e-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.831463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.831428 4911 generic.go:356] "Generic (PLEG): container finished" podID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerID="1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34" exitCode=0 Nov 01 16:22:06.831847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.831488 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" Nov 01 16:22:06.831847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.831510 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" event={"ID":"7150a085-b0d4-46a0-8a41-cc510b4ba44e","Type":"ContainerDied","Data":"1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34"} Nov 01 16:22:06.831847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.831600 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2" event={"ID":"7150a085-b0d4-46a0-8a41-cc510b4ba44e","Type":"ContainerDied","Data":"a44a084253562c5bddb546984ef8164633103901f826298dbaa4a2aad7ceaef8"} Nov 01 16:22:06.831847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.831624 4911 scope.go:122] "RemoveContainer" containerID="1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34" Nov 01 16:22:06.833264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.833235 4911 generic.go:356] "Generic (PLEG): container finished" podID="869649a0-d0cd-4236-ab36-567fc7890712" containerID="3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2" exitCode=0 Nov 01 16:22:06.833371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.833304 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" Nov 01 16:22:06.833371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.833332 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" event={"ID":"869649a0-d0cd-4236-ab36-567fc7890712","Type":"ContainerDied","Data":"3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2"} Nov 01 16:22:06.833483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.833391 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2" event={"ID":"869649a0-d0cd-4236-ab36-567fc7890712","Type":"ContainerDied","Data":"432840ce639693f6d2068ecb72d986fea03e9c35a533bdfa027071edee3a461b"} Nov 01 16:22:06.845321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.845297 4911 scope.go:122] "RemoveContainer" containerID="1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34" Nov 01 16:22:06.845655 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:22:06.845624 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34\": container with ID starting with 1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34 not found: ID does not exist" containerID="1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34" Nov 01 16:22:06.845774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.845658 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34"} err="failed to get container status \"1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34\": rpc error: code = NotFound desc = could not find container \"1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34\": container with ID starting with 1a85574cf7cb63b3c3a2cd8f914446bf25cb5fc5db7b5df9657e65fb44a06c34 not found: ID does not exist" Nov 01 16:22:06.845774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.845680 4911 scope.go:122] "RemoveContainer" containerID="3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2" Nov 01 16:22:06.859530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.859504 4911 scope.go:122] "RemoveContainer" containerID="3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2" Nov 01 16:22:06.859869 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:22:06.859843 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2\": container with ID starting with 3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2 not found: ID does not exist" containerID="3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2" Nov 01 16:22:06.859959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.859880 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2"} err="failed to get container status \"3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2\": rpc error: code = NotFound desc = could not find container \"3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2\": container with ID starting with 3e2579c86d48b987c9579d8687ba685b9076fb76b82935f21246748b71b04ae2 not found: ID does not exist" Nov 01 16:22:06.867147 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.867064 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2"] Nov 01 16:22:06.876107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.876072 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-qcdd2"] Nov 01 16:22:06.888395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.888359 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2"] Nov 01 16:22:06.894771 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:06.894732 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-q2cv2"] Nov 01 16:22:07.970797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:07.970757 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" path="/var/lib/kubelet/pods/7150a085-b0d4-46a0-8a41-cc510b4ba44e/volumes" Nov 01 16:22:07.971548 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:07.971520 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869649a0-d0cd-4236-ab36-567fc7890712" path="/var/lib/kubelet/pods/869649a0-d0cd-4236-ab36-567fc7890712/volumes" Nov 01 16:22:08.687836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:08.687803 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:22:08.700708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:08.700675 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:22:08.865962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:08.865925 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:22:10.868971 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:22:10.868935 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:22:27.487744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:27.487707 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg"] Nov 01 16:22:27.502536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:27.502359 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6"] Nov 01 16:22:32.488899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.488850 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" podUID="71f7e203-4500-4fc7-a45b-166db555f0eb" containerName="payload-processing" containerID="cri-o://0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514" gracePeriod=25 Nov 01 16:22:32.503582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.503511 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" podUID="3eb9a3c2-532a-45f4-9489-b147b2969fab" containerName="payload-pre-processing" containerID="cri-o://cc6b4b3fdff81f563c15af6ca0fc93647e610818c9426d872a5ff1bf4c5292db" gracePeriod=25 Nov 01 16:22:32.567396 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.567357 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h"] Nov 01 16:22:32.568512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.568489 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" Nov 01 16:22:32.568578 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.568517 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" Nov 01 16:22:32.568618 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.568597 4911 cpu_manager.go:414] "containerMap: removing container" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" Nov 01 16:22:32.568618 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.568605 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" Nov 01 16:22:32.568802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.568786 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7150a085-b0d4-46a0-8a41-cc510b4ba44e" containerName="istio-proxy" Nov 01 16:22:32.568840 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.568823 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="869649a0-d0cd-4236-ab36-567fc7890712" containerName="istio-proxy" Nov 01 16:22:32.573431 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.573403 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.608167 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.608131 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h"] Nov 01 16:22:32.665300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665245 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5061671b-c676-4375-b9c5-53637897fb5f-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.665496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665330 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.665496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665424 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.665496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665451 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thm9\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-kube-api-access-2thm9\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.665688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665604 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.665688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665650 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.665821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665799 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.665914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.665841 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5061671b-c676-4375-b9c5-53637897fb5f-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.666023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.666002 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.767733 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.767623 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.767733 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.767683 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5061671b-c676-4375-b9c5-53637897fb5f-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.767980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.767813 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.767980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.767879 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5061671b-c676-4375-b9c5-53637897fb5f-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.767980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.767933 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768075 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768104 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thm9\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-kube-api-access-2thm9\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768225 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768253 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768409 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768703 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768827 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768804 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5061671b-c676-4375-b9c5-53637897fb5f-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.768894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.768869 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.769117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.769096 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.771668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.771642 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.772313 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.772284 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5061671b-c676-4375-b9c5-53637897fb5f-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.777563 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.777515 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.777973 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.777955 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thm9\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-kube-api-access-2thm9\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.824979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.824951 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:22:32.896526 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.896491 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:32.971187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.971146 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f7e203-4500-4fc7-a45b-166db555f0eb-kube-api-access-fblj7\" (UniqueName: \"kubernetes.io/projected/71f7e203-4500-4fc7-a45b-166db555f0eb-kube-api-access-fblj7\") pod \"71f7e203-4500-4fc7-a45b-166db555f0eb\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " Nov 01 16:22:32.971437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.971273 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f7e203-4500-4fc7-a45b-166db555f0eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7e203-4500-4fc7-a45b-166db555f0eb-plugins-config-volume\") pod \"71f7e203-4500-4fc7-a45b-166db555f0eb\" (UID: \"71f7e203-4500-4fc7-a45b-166db555f0eb\") " Nov 01 16:22:32.971717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.971694 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f7e203-4500-4fc7-a45b-166db555f0eb-plugins-config-volume" pod "71f7e203-4500-4fc7-a45b-166db555f0eb" (UID: "71f7e203-4500-4fc7-a45b-166db555f0eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:32.971893 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.971871 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7e203-4500-4fc7-a45b-166db555f0eb-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:32.974418 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:32.974023 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f7e203-4500-4fc7-a45b-166db555f0eb-kube-api-access-fblj7" pod "71f7e203-4500-4fc7-a45b-166db555f0eb" (UID: "71f7e203-4500-4fc7-a45b-166db555f0eb"). InnerVolumeSpecName "kube-api-access-fblj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:33.074014 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.073910 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblj7\" (UniqueName: \"kubernetes.io/projected/71f7e203-4500-4fc7-a45b-166db555f0eb-kube-api-access-fblj7\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:33.137374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.137340 4911 generic.go:356] "Generic (PLEG): container finished" podID="71f7e203-4500-4fc7-a45b-166db555f0eb" containerID="0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514" exitCode=0 Nov 01 16:22:33.137541 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.137386 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" Nov 01 16:22:33.137541 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.137402 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" event={"ID":"71f7e203-4500-4fc7-a45b-166db555f0eb","Type":"ContainerDied","Data":"0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514"} Nov 01 16:22:33.137541 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.137461 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg" event={"ID":"71f7e203-4500-4fc7-a45b-166db555f0eb","Type":"ContainerDied","Data":"5cc9df442657d0e738a952f6474423717f10a395bb2c25866fe31c1bc04f1958"} Nov 01 16:22:33.137541 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.137484 4911 scope.go:122] "RemoveContainer" containerID="0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514" Nov 01 16:22:33.149351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.149331 4911 scope.go:122] "RemoveContainer" containerID="0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514" Nov 01 16:22:33.149618 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:22:33.149596 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514\": container with ID starting with 0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514 not found: ID does not exist" containerID="0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514" Nov 01 16:22:33.149670 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.149628 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514"} err="failed to get container status \"0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514\": rpc error: code = NotFound desc = could not find container \"0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514\": container with ID starting with 0abd568e61e9cba696b7188f791e59ad327b22a0579591b6ee57094e1af3e514 not found: ID does not exist" Nov 01 16:22:33.165673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.165567 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg"] Nov 01 16:22:33.169784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.169764 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0c091d-7d79b764b9-cnmxg"] Nov 01 16:22:33.212092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.212064 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h"] Nov 01 16:22:33.212593 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:22:33.212572 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5061671b_c676_4375_b9c5_53637897fb5f.slice/crio-a51a96306e1f5ebbbedb670c2998ece3c48a072eddfde35d47f3ea37eba9474a WatchSource:0}: Error finding container a51a96306e1f5ebbbedb670c2998ece3c48a072eddfde35d47f3ea37eba9474a: Status 404 returned error can't find the container with id a51a96306e1f5ebbbedb670c2998ece3c48a072eddfde35d47f3ea37eba9474a Nov 01 16:22:33.215347 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.215315 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:33.215420 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.215379 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:33.215420 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.215411 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:33.975478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:33.975397 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f7e203-4500-4fc7-a45b-166db555f0eb" path="/var/lib/kubelet/pods/71f7e203-4500-4fc7-a45b-166db555f0eb/volumes" Nov 01 16:22:34.158394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:34.158347 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" event={"ID":"5061671b-c676-4375-b9c5-53637897fb5f","Type":"ContainerStarted","Data":"4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229"} Nov 01 16:22:34.158594 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:34.158433 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" event={"ID":"5061671b-c676-4375-b9c5-53637897fb5f","Type":"ContainerStarted","Data":"a51a96306e1f5ebbbedb670c2998ece3c48a072eddfde35d47f3ea37eba9474a"} Nov 01 16:22:34.907454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:34.907405 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:34.918074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:34.910608 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:22:34.918074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:34.910675 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 16:22:35.899575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:35.899527 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:22:35.901569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:35.901522 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 16:22:36.897349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:36.897309 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:22:36.897567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:36.897371 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 16:22:37.917059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:37.902640 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:22:37.917059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:37.902709 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 16:22:38.906063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:38.903184 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:22:38.906063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:38.903254 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 16:22:39.899529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:39.899479 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:22:39.900178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:39.900128 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 16:22:40.189115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:40.188126 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" podUID="3eb9a3c2-532a-45f4-9489-b147b2969fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:22:40.634052 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:40.634004 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:22:40.635754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:40.635725 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:22:41.150490 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:22:41.150438 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:22:41.525639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:41.525604 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:41.527165 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:41.527138 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:41.528410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:41.528383 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:22:41.554878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:41.554813 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podStartSLOduration=9.554793549 podStartE2EDuration="9.554793549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:34.187684352 +0000 UTC m=+2094.332138990" watchObservedRunningTime="2026-11-01 16:22:41.554793549 +0000 UTC m=+2101.699248191" Nov 01 16:22:46.771875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.771840 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch"] Nov 01 16:22:46.772980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.772865 4911 cpu_manager.go:414] "containerMap: removing container" podUID="71f7e203-4500-4fc7-a45b-166db555f0eb" containerName="payload-processing" Nov 01 16:22:46.773188 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.773165 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f7e203-4500-4fc7-a45b-166db555f0eb" containerName="payload-processing" Nov 01 16:22:46.773568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.773544 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f7e203-4500-4fc7-a45b-166db555f0eb" containerName="payload-processing" Nov 01 16:22:46.803569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.803508 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch"] Nov 01 16:22:46.803953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.803875 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.903010 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.902976 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7225e5-openshift-default-dockercfg-bgkdt\"" Nov 01 16:22:46.913564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.913513 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.913735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.913590 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.913735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.913679 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.913846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.913825 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.913894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.913880 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.913953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.913937 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.914095 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.914075 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49vk\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-kube-api-access-z49vk\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.914157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.914109 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:46.914219 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:46.914202 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9192cc03-71c8-4c48-a288-f7d4caefd426-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.015926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.015888 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016030 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016075 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016159 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016224 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016264 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016390 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49vk\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-kube-api-access-z49vk\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016420 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016524 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9192cc03-71c8-4c48-a288-f7d4caefd426-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016632 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016582 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016632 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016613 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.016746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.016668 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.017205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.017182 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.017593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.017569 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9192cc03-71c8-4c48-a288-f7d4caefd426-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.018926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.018900 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.020000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.019971 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.025531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.025472 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49vk\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-kube-api-access-z49vk\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.025625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.025607 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.145659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.145613 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:47.459862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.459835 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch"] Nov 01 16:22:47.460497 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:22:47.460470 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9192cc03_71c8_4c48_a288_f7d4caefd426.slice/crio-c532efebbecb243806f045f1f912985f669179dd25e2f2bda96833da39178b9b WatchSource:0}: Error finding container c532efebbecb243806f045f1f912985f669179dd25e2f2bda96833da39178b9b: Status 404 returned error can't find the container with id c532efebbecb243806f045f1f912985f669179dd25e2f2bda96833da39178b9b Nov 01 16:22:47.463575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.463541 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.463660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.463612 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.463660 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.463638 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.477294 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.477268 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" event={"ID":"9192cc03-71c8-4c48-a288-f7d4caefd426","Type":"ContainerStarted","Data":"c532efebbecb243806f045f1f912985f669179dd25e2f2bda96833da39178b9b"} Nov 01 16:22:47.479136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.479107 4911 generic.go:356] "Generic (PLEG): container finished" podID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerID="21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c" exitCode=0 Nov 01 16:22:47.479253 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.479181 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" event={"ID":"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe","Type":"ContainerDied","Data":"21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c"} Nov 01 16:22:47.479732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.479711 4911 scope.go:122] "RemoveContainer" containerID="21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c" Nov 01 16:22:47.482726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.482696 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.482793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.482766 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.482844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:47.482796 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:48.488494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:48.488453 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" event={"ID":"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe","Type":"ContainerStarted","Data":"be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b"} Nov 01 16:22:48.490970 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:48.490936 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" event={"ID":"9192cc03-71c8-4c48-a288-f7d4caefd426","Type":"ContainerStarted","Data":"d186c1b73b87c4cc72b884f91cf2ecc53443f19f63c298d275c2d37680c46ebe"} Nov 01 16:22:48.529458 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:48.529407 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podStartSLOduration=2.52939294 podStartE2EDuration="2.52939294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:48.526851564 +0000 UTC m=+2108.671306207" watchObservedRunningTime="2026-11-01 16:22:48.52939294 +0000 UTC m=+2108.673847586" Nov 01 16:22:49.015103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:49.012379 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:22:49.015103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:49.012512 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 16:22:49.015103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:49.012553 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 16:22:49.146143 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:49.146107 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:49.152187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:49.151748 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 01 16:22:49.152187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:49.151815 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 01 16:22:50.017685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:50.011875 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 16:22:50.017685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:50.011937 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 16:22:50.146737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:50.146698 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 01 16:22:50.146933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:50.146771 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 01 16:22:50.195178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:50.195132 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" podUID="3eb9a3c2-532a-45f4-9489-b147b2969fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:22:51.014680 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:51.013505 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 16:22:51.014680 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:51.013568 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 16:22:51.146682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:51.146632 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 01 16:22:51.147234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:51.146696 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 01 16:22:52.011944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:52.011895 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 16:22:52.012172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:52.011962 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 16:22:52.146272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:52.146229 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 01 16:22:52.146487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:52.146289 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 01 16:22:52.630842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:52.630615 4911 generic.go:356] "Generic (PLEG): container finished" podID="3eb9a3c2-532a-45f4-9489-b147b2969fab" containerID="cc6b4b3fdff81f563c15af6ca0fc93647e610818c9426d872a5ff1bf4c5292db" exitCode=0 Nov 01 16:22:52.630842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:52.630783 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" event={"ID":"3eb9a3c2-532a-45f4-9489-b147b2969fab","Type":"ContainerDied","Data":"cc6b4b3fdff81f563c15af6ca0fc93647e610818c9426d872a5ff1bf4c5292db"} Nov 01 16:22:53.012179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.012146 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 16:22:53.012356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.012202 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 16:22:53.019510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.019455 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:22:53.130521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.130489 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb9a3c2-532a-45f4-9489-b147b2969fab-kube-api-access-m2db7\" (UniqueName: \"kubernetes.io/projected/3eb9a3c2-532a-45f4-9489-b147b2969fab-kube-api-access-m2db7\") pod \"3eb9a3c2-532a-45f4-9489-b147b2969fab\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " Nov 01 16:22:53.131272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.131247 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb9a3c2-532a-45f4-9489-b147b2969fab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9a3c2-532a-45f4-9489-b147b2969fab-plugins-config-volume\") pod \"3eb9a3c2-532a-45f4-9489-b147b2969fab\" (UID: \"3eb9a3c2-532a-45f4-9489-b147b2969fab\") " Nov 01 16:22:53.131624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.131597 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb9a3c2-532a-45f4-9489-b147b2969fab-plugins-config-volume" pod "3eb9a3c2-532a-45f4-9489-b147b2969fab" (UID: "3eb9a3c2-532a-45f4-9489-b147b2969fab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:53.132023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.132006 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9a3c2-532a-45f4-9489-b147b2969fab-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:53.133133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.133095 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb9a3c2-532a-45f4-9489-b147b2969fab-kube-api-access-m2db7" pod "3eb9a3c2-532a-45f4-9489-b147b2969fab" (UID: "3eb9a3c2-532a-45f4-9489-b147b2969fab"). InnerVolumeSpecName "kube-api-access-m2db7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:53.147275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.147085 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 01 16:22:53.147275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.147202 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 01 16:22:53.233182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.233035 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2db7\" (UniqueName: \"kubernetes.io/projected/3eb9a3c2-532a-45f4-9489-b147b2969fab-kube-api-access-m2db7\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:53.641931 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.641901 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" Nov 01 16:22:53.642357 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.641940 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6" event={"ID":"3eb9a3c2-532a-45f4-9489-b147b2969fab","Type":"ContainerDied","Data":"bfd6f03b98a1321284248cd03392cfa75b9a006b81454593383631fdab8babbf"} Nov 01 16:22:53.642357 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.642007 4911 scope.go:122] "RemoveContainer" containerID="cc6b4b3fdff81f563c15af6ca0fc93647e610818c9426d872a5ff1bf4c5292db" Nov 01 16:22:53.710180 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.710128 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6"] Nov 01 16:22:53.715490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:53.715463 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0c091d-7fb787cc8c-658b6"] Nov 01 16:22:54.011758 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:54.011623 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 01 16:22:54.011971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:54.011773 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 01 16:22:54.014234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:54.013810 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb9a3c2-532a-45f4-9489-b147b2969fab" path="/var/lib/kubelet/pods/3eb9a3c2-532a-45f4-9489-b147b2969fab/volumes" Nov 01 16:22:54.146863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:54.146832 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 01 16:22:54.147029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:54.146880 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 01 16:22:55.144856 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:55.144820 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:22:55.145394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:55.145341 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:22:55.146346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:55.146326 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:22:55.152153 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:55.152127 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:55.152626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:55.152565 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:22:55.153298 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:22:55.153277 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:23:10.838429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:10.838385 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz"] Nov 01 16:23:10.839702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:10.839471 4911 cpu_manager.go:414] "containerMap: removing container" podUID="3eb9a3c2-532a-45f4-9489-b147b2969fab" containerName="payload-pre-processing" Nov 01 16:23:10.839702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:10.839500 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb9a3c2-532a-45f4-9489-b147b2969fab" containerName="payload-pre-processing" Nov 01 16:23:10.839905 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:10.839761 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb9a3c2-532a-45f4-9489-b147b2969fab" containerName="payload-pre-processing" Nov 01 16:23:10.856904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:10.856749 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz"] Nov 01 16:23:10.857178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:10.856870 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:10.876687 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:10.876656 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5e2d03-openshift-default-dockercfg-7zm9b\"" Nov 01 16:23:10.881301 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:23:10.881262 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:23:11.003238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003198 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68xc\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-kube-api-access-w68xc\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003338 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-podinfo\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003367 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-certs\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003391 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-data\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003515 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-credential-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003676 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-envoy\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003763 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003713 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-token\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003763 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003747 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istiod-ca-cert\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.003878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.003824 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.105832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.105734 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-podinfo\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.105832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.105792 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-certs\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.105902 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-data\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.105988 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-credential-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106205 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-envoy\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106228 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-token\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106243 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-certs\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106249 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istiod-ca-cert\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106297 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-data\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106335 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106393 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-credential-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.106635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.106565 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68xc\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-kube-api-access-w68xc\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.107032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.107006 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-socket\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.107263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.107245 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istiod-ca-cert\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.108830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.108804 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-podinfo\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.109023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.109006 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-envoy\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.115013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.114988 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-token\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.115132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.115113 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68xc\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-kube-api-access-w68xc\") pod \"e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.199553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.199507 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:11.514640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.514606 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz"] Nov 01 16:23:11.515085 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:23:11.515019 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f27b0b4_d63f_45c4_8408_f67a0a6eaf81.slice/crio-28c668c23213df1d59b29747e3f1b0fcfb6165c03316cc67b8e29f6fb590f23f WatchSource:0}: Error finding container 28c668c23213df1d59b29747e3f1b0fcfb6165c03316cc67b8e29f6fb590f23f: Status 404 returned error can't find the container with id 28c668c23213df1d59b29747e3f1b0fcfb6165c03316cc67b8e29f6fb590f23f Nov 01 16:23:11.518424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.518384 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:11.518502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.518473 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:11.518549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.518518 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:11.811482 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.811376 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" event={"ID":"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81","Type":"ContainerStarted","Data":"c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7"} Nov 01 16:23:11.811482 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.811471 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" event={"ID":"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81","Type":"ContainerStarted","Data":"28c668c23213df1d59b29747e3f1b0fcfb6165c03316cc67b8e29f6fb590f23f"} Nov 01 16:23:11.835503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:11.835448 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podStartSLOduration=1.835431523 podStartE2EDuration="1.835431523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:11.831009867 +0000 UTC m=+2131.975464508" watchObservedRunningTime="2026-11-01 16:23:11.835431523 +0000 UTC m=+2131.979886164" Nov 01 16:23:12.200133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:12.200019 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:12.201494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:12.201464 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 16:23:12.201635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:12.201515 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 16:23:13.200169 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:13.200130 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 16:23:13.200559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:13.200221 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 16:23:14.200852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:14.200813 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 16:23:14.201426 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:14.200877 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 16:23:15.203096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:15.200168 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 16:23:15.203096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:15.200246 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 16:23:16.200382 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:16.200341 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 16:23:16.200570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:16.200401 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 16:23:17.200604 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:17.200565 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 01 16:23:17.201013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:17.200641 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 01 16:23:18.296915 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:18.296877 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:18.297983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:18.297950 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:18.304319 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:18.304290 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:23:31.970463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:31.970431 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l"] Nov 01 16:23:31.976263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:31.976238 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.018134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.018098 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l"] Nov 01 16:23:32.041163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041127 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041168 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c3e2b6-d2d0-428f-907a-aa53f210f559-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041194 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041236 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041264 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041361 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041442 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041493 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.041720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.041574 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkqs\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-kube-api-access-vrkqs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.142615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.142582 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.142816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.142642 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.142816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.142717 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkqs\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-kube-api-access-vrkqs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.142969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.142874 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.142969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.142902 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c3e2b6-d2d0-428f-907a-aa53f210f559-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.142969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.142930 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.143203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.142978 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.143203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.143008 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.143203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.143110 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.143203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.143126 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.143434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.143332 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.143434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.143393 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.143638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.143609 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.144153 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.144120 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c3e2b6-d2d0-428f-907a-aa53f210f559-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.146175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.146149 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.146506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.146488 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.152513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.152489 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.152640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.152593 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkqs\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-kube-api-access-vrkqs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.291854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.291816 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:32.673619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.671950 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:32.673619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.672110 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:32.673619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.672160 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:32.674535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:32.674412 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l"] Nov 01 16:23:33.056716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:33.056673 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" event={"ID":"21c3e2b6-d2d0-428f-907a-aa53f210f559","Type":"ContainerStarted","Data":"7f762e6ff816dd2c766d8c363312e6cc002c55c7f5fc94473f03c925956f5981"} Nov 01 16:23:33.057320 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:33.056742 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" event={"ID":"21c3e2b6-d2d0-428f-907a-aa53f210f559","Type":"ContainerStarted","Data":"2febe53a16c99361fd3b8701091f27bea46303b4e64cd6bb0201c077120f865b"} Nov 01 16:23:33.083752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:33.083688 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podStartSLOduration=2.083669725 podStartE2EDuration="2.083669725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:33.07511265 +0000 UTC m=+2153.219567293" watchObservedRunningTime="2026-11-01 16:23:33.083669725 +0000 UTC m=+2153.228124368" Nov 01 16:23:33.292745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:33.292708 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:33.293976 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:33.293937 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:23:33.294131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:33.293992 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 01 16:23:34.292446 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:34.292387 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:23:34.293088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:34.293007 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 01 16:23:35.292725 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:35.292681 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:23:35.293372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:35.293325 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 01 16:23:36.292997 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:36.292949 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:23:36.293507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:36.293021 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 01 16:23:37.292503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:37.292454 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:23:37.292710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:37.292524 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 01 16:23:38.293060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:38.292625 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:23:38.293060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:38.292689 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 01 16:23:39.292480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:39.292437 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:23:39.292775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:39.292739 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 01 16:23:40.300063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:40.299811 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:40.300525 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:40.300394 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:40.301216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:40.301195 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:23:40.869814 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:23:40.869777 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:23:41.373106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.373070 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j"] Nov 01 16:23:41.378330 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.378301 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.393640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.393604 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j"] Nov 01 16:23:41.428957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.425234 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv"] Nov 01 16:23:41.452493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.452449 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.483929 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484085 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484198 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484268 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484380 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484464 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484605 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdkj\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-kube-api-access-jpdkj\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484667 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.500142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.484701 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.521710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.521606 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv"] Nov 01 16:23:41.586428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.586390 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.587243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.587203 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.592836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.589483 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.592446 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.592807 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593098 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593160 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593208 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jvq\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-kube-api-access-l8jvq\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593265 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593323 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593379 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593399 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593416 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593460 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593548 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593574 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392c10c3-f485-4701-b2f0-5de1966b35bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593597 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593625 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.593783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593655 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.593783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.593696 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdkj\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-kube-api-access-jpdkj\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.594266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.594240 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.594703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.594681 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.594838 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.594778 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.595220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.595193 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.596841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.596804 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.672064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.671962 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdkj\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-kube-api-access-jpdkj\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.684886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.684851 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698110 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392c10c3-f485-4701-b2f0-5de1966b35bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698167 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698212 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698344 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698424 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698468 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jvq\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-kube-api-access-l8jvq\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698545 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698627 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.715558 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.698678 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.728849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.727369 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.728849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.728396 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.729383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.729344 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392c10c3-f485-4701-b2f0-5de1966b35bf-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.730351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.730322 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.734725 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.734441 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.734862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.734754 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.745242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.743702 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.745242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.744491 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:41.766300 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.766263 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jvq\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-kube-api-access-l8jvq\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.785516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.785480 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:41.819208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:41.819129 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:42.098280 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.098236 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.098531 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.098508 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.098677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.098660 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.100258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.100234 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j"] Nov 01 16:23:42.166790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.166745 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.166918 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.166839 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.166918 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.166879 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.169189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.169159 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv"] Nov 01 16:23:42.196158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.196119 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" event={"ID":"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2","Type":"ContainerStarted","Data":"477f23a028168a46c626f156157abc3d5df780a3853f14031734128e045ad3f1"} Nov 01 16:23:42.197880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:42.197844 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" event={"ID":"392c10c3-f485-4701-b2f0-5de1966b35bf","Type":"ContainerStarted","Data":"6df980a57657b58672b6a0ec0954a24abf5a2921ce667cdae1270bc65f8eb079"} Nov 01 16:23:43.238835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.237528 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" event={"ID":"392c10c3-f485-4701-b2f0-5de1966b35bf","Type":"ContainerStarted","Data":"71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587"} Nov 01 16:23:43.242262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.242221 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" event={"ID":"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2","Type":"ContainerStarted","Data":"1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6"} Nov 01 16:23:43.296063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.294362 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podStartSLOduration=2.2943432120000002 podStartE2EDuration="2.294343212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:43.28695571 +0000 UTC m=+2163.431410354" watchObservedRunningTime="2026-11-01 16:23:43.294343212 +0000 UTC m=+2163.438797855" Nov 01 16:23:43.766068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.756978 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 16:23:43.766068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.757071 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 16:23:43.766068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.757401 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:43.823983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.823891 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:43.824325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.824296 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:23:43.824491 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:43.824462 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 16:23:44.219311 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.219204 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:44.345248 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.345196 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:44.475241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.475130 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:44.706064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.705125 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" podUID="4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:44.739086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.738650 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:44.763879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.763839 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 16:23:44.765936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.765899 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 16:23:44.820487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.820445 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:23:44.820777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.820739 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 16:23:44.862166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.858160 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.172:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:44.940065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.939151 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:44.995141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.994238 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:45.019066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:44.999100 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" podUID="fb7b581d-c541-401b-a814-81bdc2737087" containerName="main" probeResult="failure" output="Get \"https://10.132.0.166:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:23:45.752425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:45.752383 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 16:23:45.752993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:45.752962 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 16:23:45.819655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:45.819609 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:23:45.819855 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:45.819672 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 16:23:46.748944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:46.748900 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 16:23:46.749208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:46.748978 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 16:23:46.821066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:46.820925 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:23:46.821587 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:46.821020 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 16:23:47.749487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:47.749429 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 16:23:47.749811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:47.749766 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 16:23:47.912066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:47.911170 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:23:47.912066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:47.911241 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 16:23:48.771064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:48.770532 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 16:23:48.771064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:48.770599 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 16:23:48.820453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:48.820311 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 16:23:48.820453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:48.820371 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 16:23:49.826019 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:49.825979 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:49.826592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:49.826572 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:49.827516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:49.827495 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:23:49.845408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:49.845376 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:49.845933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:49.845904 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:49.850263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:49.850241 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:23:49.852214 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:49.852169 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podStartSLOduration=8.852157863 podStartE2EDuration="8.852157863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:43.317388112 +0000 UTC m=+2163.461842764" watchObservedRunningTime="2026-11-01 16:23:49.852157863 +0000 UTC m=+2169.996612529" Nov 01 16:23:59.424457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:59.424417 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6"] Nov 01 16:23:59.812582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:23:59.812533 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" podUID="7b92acb9-5d34-4f35-9926-cdb0334f5c80" containerName="maas-api" containerID="cri-o://d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3" gracePeriod=30 Nov 01 16:24:00.249392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.249360 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:24:00.358686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.358597 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b92acb9-5d34-4f35-9926-cdb0334f5c80-kube-api-access-88j2n\" (UniqueName: \"kubernetes.io/projected/7b92acb9-5d34-4f35-9926-cdb0334f5c80-kube-api-access-88j2n\") pod \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " Nov 01 16:24:00.358686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.358676 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b92acb9-5d34-4f35-9926-cdb0334f5c80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b92acb9-5d34-4f35-9926-cdb0334f5c80-openshift-service-ca\") pod \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " Nov 01 16:24:00.358985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.358960 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-maas-api-tls\") pod \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " Nov 01 16:24:00.359161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.359007 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-metrics-certs\") pod \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\" (UID: \"7b92acb9-5d34-4f35-9926-cdb0334f5c80\") " Nov 01 16:24:00.359161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.359105 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b92acb9-5d34-4f35-9926-cdb0334f5c80-openshift-service-ca" pod "7b92acb9-5d34-4f35-9926-cdb0334f5c80" (UID: "7b92acb9-5d34-4f35-9926-cdb0334f5c80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:00.359920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.359867 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b92acb9-5d34-4f35-9926-cdb0334f5c80-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:00.361507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.361453 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-maas-api-tls" pod "7b92acb9-5d34-4f35-9926-cdb0334f5c80" (UID: "7b92acb9-5d34-4f35-9926-cdb0334f5c80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:24:00.361824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.361774 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b92acb9-5d34-4f35-9926-cdb0334f5c80-kube-api-access-88j2n" pod "7b92acb9-5d34-4f35-9926-cdb0334f5c80" (UID: "7b92acb9-5d34-4f35-9926-cdb0334f5c80"). InnerVolumeSpecName "kube-api-access-88j2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:00.362258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.362231 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-metrics-certs" pod "7b92acb9-5d34-4f35-9926-cdb0334f5c80" (UID: "7b92acb9-5d34-4f35-9926-cdb0334f5c80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:24:00.461785 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.461748 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:00.461785 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.461781 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b92acb9-5d34-4f35-9926-cdb0334f5c80-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:00.461785 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.461791 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88j2n\" (UniqueName: \"kubernetes.io/projected/7b92acb9-5d34-4f35-9926-cdb0334f5c80-kube-api-access-88j2n\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:00.813444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.813411 4911 generic.go:356] "Generic (PLEG): container finished" podID="7b92acb9-5d34-4f35-9926-cdb0334f5c80" containerID="d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3" exitCode=0 Nov 01 16:24:00.813631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.813452 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" Nov 01 16:24:00.813631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.813491 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" event={"ID":"7b92acb9-5d34-4f35-9926-cdb0334f5c80","Type":"ContainerDied","Data":"d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3"} Nov 01 16:24:00.813631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.813549 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6" event={"ID":"7b92acb9-5d34-4f35-9926-cdb0334f5c80","Type":"ContainerDied","Data":"80e615439181cb4910f8224ce16356fd430ef4ac4f0814b0b8049ecf0d5db4bd"} Nov 01 16:24:00.813631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.813577 4911 scope.go:122] "RemoveContainer" containerID="d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3" Nov 01 16:24:00.826265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.826092 4911 scope.go:122] "RemoveContainer" containerID="d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3" Nov 01 16:24:00.826439 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:24:00.826414 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3\": container with ID starting with d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3 not found: ID does not exist" containerID="d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3" Nov 01 16:24:00.826494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.826449 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3"} err="failed to get container status \"d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3\": rpc error: code = NotFound desc = could not find container \"d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3\": container with ID starting with d4eafc1c2ee2d821666ca57b65f4d59333ec3d2d87be8b77602c783151ee62b3 not found: ID does not exist" Nov 01 16:24:00.840499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.840474 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6"] Nov 01 16:24:00.844741 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:00.844713 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e044c-8b9f5d995-w92x6"] Nov 01 16:24:01.971230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:01.971192 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b92acb9-5d34-4f35-9926-cdb0334f5c80" path="/var/lib/kubelet/pods/7b92acb9-5d34-4f35-9926-cdb0334f5c80/volumes" Nov 01 16:24:06.469124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:06.468524 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv"] Nov 01 16:24:06.469124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:06.468786 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" containerID="cri-o://37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe" gracePeriod=30 Nov 01 16:24:09.087824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:09.087789 4911 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:24:09.088310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:09.087865 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:24:10.870086 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:24:10.869996 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:24:11.773615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.773578 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:24:11.887581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887548 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-podinfo\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.887995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887602 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-certs\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.887995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887661 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-socket\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.887995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887731 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-envoy\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.887995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887835 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-token\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-token\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.887995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887907 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-data\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.887995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887973 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-kube-api-access-7wc88\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-kube-api-access-7wc88\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.888365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.887999 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istiod-ca-cert\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.888365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888009 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-certs" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:11.888365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888033 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-credential-socket\") pod \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\" (UID: \"307adf41-f7d6-4b51-a0b4-a0bfb1c92728\") " Nov 01 16:24:11.888365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888027 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-socket" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:11.888365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888285 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-data" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:11.888658 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888494 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-credential-socket" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:11.888658 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888434 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istiod-ca-cert" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:11.888965 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888941 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.888965 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888970 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.889144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.888986 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.889144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.889002 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.889144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.889016 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.890030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.889996 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-podinfo" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:24:11.890577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.890553 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-envoy" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:11.890656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.890591 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-kube-api-access-7wc88" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "kube-api-access-7wc88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:11.890656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.890614 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-token" pod "307adf41-f7d6-4b51-a0b4-a0bfb1c92728" (UID: "307adf41-f7d6-4b51-a0b4-a0bfb1c92728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:11.990642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.990607 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.990642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.990644 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.990849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.990662 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wc88\" (UniqueName: \"kubernetes.io/projected/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-kube-api-access-7wc88\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.990849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:11.990681 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307adf41-f7d6-4b51-a0b4-a0bfb1c92728-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:12.038758 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.038647 4911 generic.go:356] "Generic (PLEG): container finished" podID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerID="37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe" exitCode=0 Nov 01 16:24:12.038934 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.038888 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" event={"ID":"307adf41-f7d6-4b51-a0b4-a0bfb1c92728","Type":"ContainerDied","Data":"37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe"} Nov 01 16:24:12.039004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.038947 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" event={"ID":"307adf41-f7d6-4b51-a0b4-a0bfb1c92728","Type":"ContainerDied","Data":"8b2dbddae426b368821e1ed382f9101d7ab779757186a18982527b222fecfce2"} Nov 01 16:24:12.039004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.038979 4911 scope.go:122] "RemoveContainer" containerID="37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe" Nov 01 16:24:12.039235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.039206 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv" Nov 01 16:24:12.055712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.055687 4911 scope.go:122] "RemoveContainer" containerID="37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe" Nov 01 16:24:12.056156 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:24:12.056120 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe\": container with ID starting with 37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe not found: ID does not exist" containerID="37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe" Nov 01 16:24:12.056270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.056158 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe"} err="failed to get container status \"37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe\": rpc error: code = NotFound desc = could not find container \"37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe\": container with ID starting with 37c03092b6f00a9ce71f89dbca38e0ac182c5270db2080d4656dffb5f976acfe not found: ID does not exist" Nov 01 16:24:12.069252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.069183 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv"] Nov 01 16:24:12.074184 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:12.074148 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-jsnbv"] Nov 01 16:24:13.969683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:13.969648 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" path="/var/lib/kubelet/pods/307adf41-f7d6-4b51-a0b4-a0bfb1c92728/volumes" Nov 01 16:24:17.129267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:17.129228 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s"] Nov 01 16:24:22.130550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:22.130457 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" podUID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" containerName="payload-pre-processing" containerID="cri-o://6d7ad9ad43324a1c0c5485006196a55631fbd5f0143de1ff46205017bb6dddd9" gracePeriod=25 Nov 01 16:24:26.141655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:26.141618 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" podUID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:24:36.164241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:36.164198 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" podUID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:24:40.869987 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:24:40.869949 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:24:42.452878 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.452840 4911 generic.go:356] "Generic (PLEG): container finished" podID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" containerID="6d7ad9ad43324a1c0c5485006196a55631fbd5f0143de1ff46205017bb6dddd9" exitCode=0 Nov 01 16:24:42.453410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.453203 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" event={"ID":"ff14d30a-77f0-4394-bfd5-bcc60bdd866f","Type":"ContainerDied","Data":"6d7ad9ad43324a1c0c5485006196a55631fbd5f0143de1ff46205017bb6dddd9"} Nov 01 16:24:42.510422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.510350 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:24:42.636600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.636564 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-plugins-config-volume\") pod \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " Nov 01 16:24:42.636816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.636749 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-kube-api-access-2jwm9\" (UniqueName: \"kubernetes.io/projected/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-kube-api-access-2jwm9\") pod \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\" (UID: \"ff14d30a-77f0-4394-bfd5-bcc60bdd866f\") " Nov 01 16:24:42.637025 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.636998 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-plugins-config-volume" pod "ff14d30a-77f0-4394-bfd5-bcc60bdd866f" (UID: "ff14d30a-77f0-4394-bfd5-bcc60bdd866f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:42.637520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.637496 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:42.639094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.639063 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-kube-api-access-2jwm9" pod "ff14d30a-77f0-4394-bfd5-bcc60bdd866f" (UID: "ff14d30a-77f0-4394-bfd5-bcc60bdd866f"). InnerVolumeSpecName "kube-api-access-2jwm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:42.739007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:42.738967 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwm9\" (UniqueName: \"kubernetes.io/projected/ff14d30a-77f0-4394-bfd5-bcc60bdd866f-kube-api-access-2jwm9\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:43.484062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:43.483587 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" event={"ID":"ff14d30a-77f0-4394-bfd5-bcc60bdd866f","Type":"ContainerDied","Data":"f29d92c1ae6411d2f7b00f1c8df6fab1b4d4c927e442b21798445455d3f98ed8"} Nov 01 16:24:43.484062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:43.483665 4911 scope.go:122] "RemoveContainer" containerID="6d7ad9ad43324a1c0c5485006196a55631fbd5f0143de1ff46205017bb6dddd9" Nov 01 16:24:43.484062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:43.483849 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s" Nov 01 16:24:43.552107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:43.551564 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s"] Nov 01 16:24:43.561923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:43.561871 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ae79df-bf79f4f59-rxs4s"] Nov 01 16:24:43.972105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:24:43.972064 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" path="/var/lib/kubelet/pods/ff14d30a-77f0-4394-bfd5-bcc60bdd866f/volumes" Nov 01 16:25:01.492980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:01.492941 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp"] Nov 01 16:25:01.493560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:01.493177 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" containerID="cri-o://5adb8de432cb3461427a34d2b42bb6901674140479af7fca8349c6b93dadd9e1" gracePeriod=30 Nov 01 16:25:06.950574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:06.943545 4911 generic.go:356] "Generic (PLEG): container finished" podID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerID="5adb8de432cb3461427a34d2b42bb6901674140479af7fca8349c6b93dadd9e1" exitCode=0 Nov 01 16:25:06.950574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:06.943785 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" event={"ID":"b8d55996-1e97-4797-ba6a-31a5ff5974c4","Type":"ContainerDied","Data":"5adb8de432cb3461427a34d2b42bb6901674140479af7fca8349c6b93dadd9e1"} Nov 01 16:25:07.020131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.020096 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 16:25:07.124932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.124899 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-kube-api-access-m6k4m\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-kube-api-access-m6k4m\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125021 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-data\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125117 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-credential-socket\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125150 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-envoy\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125185 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-token\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-token\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125226 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-podinfo\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125342 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istiod-ca-cert\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125334 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-data" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:07.125498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125413 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-credential-socket" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:07.125616 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125537 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-certs\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125674 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-socket\") pod \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\" (UID: \"b8d55996-1e97-4797-ba6a-31a5ff5974c4\") " Nov 01 16:25:07.125757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125711 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istiod-ca-cert" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:25:07.126320 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.125983 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-certs" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:07.126320 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.126013 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-socket" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:07.126787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.126671 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.126787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.126697 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.126787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.126712 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.126787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.126725 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.126787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.126738 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.128872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.127757 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-kube-api-access-m6k4m" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "kube-api-access-m6k4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:07.128872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.128514 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-envoy" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:25:07.134960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.134931 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-podinfo" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:25:07.135794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.135758 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-token" pod "b8d55996-1e97-4797-ba6a-31a5ff5974c4" (UID: "b8d55996-1e97-4797-ba6a-31a5ff5974c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:07.228823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.228733 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6k4m\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-kube-api-access-m6k4m\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.228823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.228769 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.228823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.228779 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.228823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.228789 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d55996-1e97-4797-ba6a-31a5ff5974c4-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:07.953416 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.953380 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" Nov 01 16:25:07.953927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.953411 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp" event={"ID":"b8d55996-1e97-4797-ba6a-31a5ff5974c4","Type":"ContainerDied","Data":"e6dee911609c6c3b12768ea817a32673d7be6f98da14cbb67d4a52f022924668"} Nov 01 16:25:07.953927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.953481 4911 scope.go:122] "RemoveContainer" containerID="5adb8de432cb3461427a34d2b42bb6901674140479af7fca8349c6b93dadd9e1" Nov 01 16:25:07.989611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.989582 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp"] Nov 01 16:25:07.993710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:07.993687 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-b9sqp"] Nov 01 16:25:09.985589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:09.985547 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" path="/var/lib/kubelet/pods/b8d55996-1e97-4797-ba6a-31a5ff5974c4/volumes" Nov 01 16:25:10.868894 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:25:10.868856 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:25:40.871671 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:25:40.871623 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:25:52.324788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:25:52.324550 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6"] Nov 01 16:26:07.362216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.362154 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" podUID="fb7b581d-c541-401b-a814-81bdc2737087" containerName="main" containerID="cri-o://fbcc83c894b04fc0aec917580b7662807010c4f13be5e9a4e9cbc8d9cec471c5" gracePeriod=45 Nov 01 16:26:07.632232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.632056 4911 generic.go:356] "Generic (PLEG): container finished" podID="fb7b581d-c541-401b-a814-81bdc2737087" containerID="fbcc83c894b04fc0aec917580b7662807010c4f13be5e9a4e9cbc8d9cec471c5" exitCode=0 Nov 01 16:26:07.632232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.632122 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" event={"ID":"fb7b581d-c541-401b-a814-81bdc2737087","Type":"ContainerDied","Data":"fbcc83c894b04fc0aec917580b7662807010c4f13be5e9a4e9cbc8d9cec471c5"} Nov 01 16:26:07.697817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.697784 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:26:07.757669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.757626 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-dshm\") pod \"fb7b581d-c541-401b-a814-81bdc2737087\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " Nov 01 16:26:07.757846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.757737 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-tmp-dir\") pod \"fb7b581d-c541-401b-a814-81bdc2737087\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " Nov 01 16:26:07.757846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.757804 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7b581d-c541-401b-a814-81bdc2737087-kube-api-access-j8jww\" (UniqueName: \"kubernetes.io/projected/fb7b581d-c541-401b-a814-81bdc2737087-kube-api-access-j8jww\") pod \"fb7b581d-c541-401b-a814-81bdc2737087\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " Nov 01 16:26:07.757986 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.757956 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7b581d-c541-401b-a814-81bdc2737087-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7b581d-c541-401b-a814-81bdc2737087-tls-certs\") pod \"fb7b581d-c541-401b-a814-81bdc2737087\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " Nov 01 16:26:07.758139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.758119 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-home\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-home\") pod \"fb7b581d-c541-401b-a814-81bdc2737087\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " Nov 01 16:26:07.758229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.758212 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-model-cache\") pod \"fb7b581d-c541-401b-a814-81bdc2737087\" (UID: \"fb7b581d-c541-401b-a814-81bdc2737087\") " Nov 01 16:26:07.759589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.759555 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-model-cache" pod "fb7b581d-c541-401b-a814-81bdc2737087" (UID: "fb7b581d-c541-401b-a814-81bdc2737087"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:07.759827 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.759806 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-home" pod "fb7b581d-c541-401b-a814-81bdc2737087" (UID: "fb7b581d-c541-401b-a814-81bdc2737087"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:07.760016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.759994 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-tmp-dir" pod "fb7b581d-c541-401b-a814-81bdc2737087" (UID: "fb7b581d-c541-401b-a814-81bdc2737087"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:07.762174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.762132 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7b581d-c541-401b-a814-81bdc2737087-tls-certs" pod "fb7b581d-c541-401b-a814-81bdc2737087" (UID: "fb7b581d-c541-401b-a814-81bdc2737087"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:26:07.762493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.762469 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7b581d-c541-401b-a814-81bdc2737087-kube-api-access-j8jww" pod "fb7b581d-c541-401b-a814-81bdc2737087" (UID: "fb7b581d-c541-401b-a814-81bdc2737087"). InnerVolumeSpecName "kube-api-access-j8jww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:07.763443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.763408 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-dshm" pod "fb7b581d-c541-401b-a814-81bdc2737087" (UID: "fb7b581d-c541-401b-a814-81bdc2737087"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:26:07.860703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.860666 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:07.860703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.860703 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8jww\" (UniqueName: \"kubernetes.io/projected/fb7b581d-c541-401b-a814-81bdc2737087-kube-api-access-j8jww\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:07.860926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.860718 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7b581d-c541-401b-a814-81bdc2737087-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:07.860926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.860728 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:07.860926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.860737 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:07.860926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:07.860746 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7b581d-c541-401b-a814-81bdc2737087-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:08.641593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:08.641557 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" event={"ID":"fb7b581d-c541-401b-a814-81bdc2737087","Type":"ContainerDied","Data":"0af90e17bf7682561715f8d478563b507529ce64411e3e0b128816fb032a9c02"} Nov 01 16:26:08.642055 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:08.641610 4911 scope.go:122] "RemoveContainer" containerID="fbcc83c894b04fc0aec917580b7662807010c4f13be5e9a4e9cbc8d9cec471c5" Nov 01 16:26:08.642055 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:08.641565 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6" Nov 01 16:26:08.662321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:08.662294 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6"] Nov 01 16:26:08.667486 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:08.667462 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d121/auth-test-model-c8d121-kserve-76d6cd887c-sm8d6"] Nov 01 16:26:09.967633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:09.967598 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7b581d-c541-401b-a814-81bdc2737087" path="/var/lib/kubelet/pods/fb7b581d-c541-401b-a814-81bdc2737087/volumes" Nov 01 16:26:10.904433 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:26:10.904388 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:26:34.281478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.281441 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8"] Nov 01 16:26:34.282160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282145 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" Nov 01 16:26:34.282223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282160 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" Nov 01 16:26:34.282223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282178 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7b92acb9-5d34-4f35-9926-cdb0334f5c80" containerName="maas-api" Nov 01 16:26:34.282223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282185 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b92acb9-5d34-4f35-9926-cdb0334f5c80" containerName="maas-api" Nov 01 16:26:34.282223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282193 4911 cpu_manager.go:414] "containerMap: removing container" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" Nov 01 16:26:34.282223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282198 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" Nov 01 16:26:34.282223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282217 4911 cpu_manager.go:414] "containerMap: removing container" podUID="fb7b581d-c541-401b-a814-81bdc2737087" containerName="main" Nov 01 16:26:34.282223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282221 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7b581d-c541-401b-a814-81bdc2737087" containerName="main" Nov 01 16:26:34.282443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282237 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" containerName="payload-pre-processing" Nov 01 16:26:34.282443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282242 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" containerName="payload-pre-processing" Nov 01 16:26:34.282443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282351 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7b581d-c541-401b-a814-81bdc2737087" containerName="main" Nov 01 16:26:34.282443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282359 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff14d30a-77f0-4394-bfd5-bcc60bdd866f" containerName="payload-pre-processing" Nov 01 16:26:34.282443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282372 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d55996-1e97-4797-ba6a-31a5ff5974c4" containerName="istio-proxy" Nov 01 16:26:34.282443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282382 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b92acb9-5d34-4f35-9926-cdb0334f5c80" containerName="maas-api" Nov 01 16:26:34.282443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.282388 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="307adf41-f7d6-4b51-a0b4-a0bfb1c92728" containerName="istio-proxy" Nov 01 16:26:34.287874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.287844 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.293709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.293675 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8"] Nov 01 16:26:34.348574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.348539 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b94b5-dockercfg-2fq72\"" Nov 01 16:26:34.349178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.349058 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b94b5\"" Nov 01 16:26:34.410938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.410873 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52753ffd-ccbf-4250-a1dc-989d4dad113a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.411471 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.411437 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxc4l\" (UniqueName: \"kubernetes.io/projected/52753ffd-ccbf-4250-a1dc-989d4dad113a-kube-api-access-kxc4l\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.514168 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.514122 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxc4l\" (UniqueName: \"kubernetes.io/projected/52753ffd-ccbf-4250-a1dc-989d4dad113a-kube-api-access-kxc4l\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.514384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.514320 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52753ffd-ccbf-4250-a1dc-989d4dad113a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.515909 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.515876 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52753ffd-ccbf-4250-a1dc-989d4dad113a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.547811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.547717 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxc4l\" (UniqueName: \"kubernetes.io/projected/52753ffd-ccbf-4250-a1dc-989d4dad113a-kube-api-access-kxc4l\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.607600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.607564 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:34.898403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.898369 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8"] Nov 01 16:26:34.898757 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:26:34.898733 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52753ffd_ccbf_4250_a1dc_989d4dad113a.slice/crio-ea9d8f11a06e49ea9c9ff2b5f7d987003ffb9f78bcec0163ac8f95fc86b18f2e WatchSource:0}: Error finding container ea9d8f11a06e49ea9c9ff2b5f7d987003ffb9f78bcec0163ac8f95fc86b18f2e: Status 404 returned error can't find the container with id ea9d8f11a06e49ea9c9ff2b5f7d987003ffb9f78bcec0163ac8f95fc86b18f2e Nov 01 16:26:34.901372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:34.901345 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:26:35.004682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:35.004647 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" event={"ID":"52753ffd-ccbf-4250-a1dc-989d4dad113a","Type":"ContainerStarted","Data":"ea9d8f11a06e49ea9c9ff2b5f7d987003ffb9f78bcec0163ac8f95fc86b18f2e"} Nov 01 16:26:37.022734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:37.022697 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" event={"ID":"52753ffd-ccbf-4250-a1dc-989d4dad113a","Type":"ContainerStarted","Data":"7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250"} Nov 01 16:26:37.023221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:37.022824 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:37.043824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:37.043765 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" podStartSLOduration=1.584772767 podStartE2EDuration="3.043748096s" totalImagesPullingTime="1.458975329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:37.040277024 +0000 UTC m=+2337.184731665" watchObservedRunningTime="2026-11-01 16:26:37.043748096 +0000 UTC m=+2337.188202740" Nov 01 16:26:40.868573 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:26:40.868535 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:26:45.687771 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:45.684181 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp"] Nov 01 16:26:48.033730 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:48.033695 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:26:57.052865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.052816 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4"] Nov 01 16:26:57.062635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.062608 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.066635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.066605 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4"] Nov 01 16:26:57.073286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.073257 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"kube-root-ca.crt\"" Nov 01 16:26:57.073426 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.073381 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"default-dockercfg-55w5d\"" Nov 01 16:26:57.073495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.073442 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"openshift-service-ca.crt\"" Nov 01 16:26:57.073567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.073552 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-383fde\"/\"sub-test-model-383fde-kserve-self-signed-certs\"" Nov 01 16:26:57.159704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.159657 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tls-certs\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.159924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.159749 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-model-cache\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.159924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.159886 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-home\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.160082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.160018 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkcb\" (UniqueName: \"kubernetes.io/projected/9811d4e5-fe1b-4920-9601-fc298fb4ab66-kube-api-access-nfkcb\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.160244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.160225 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-dshm\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.160293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.160278 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tmp-dir\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.265201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.265142 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tls-certs\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.265415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.265282 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-model-cache\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.265415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.265363 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-home\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.265540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.265440 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkcb\" (UniqueName: \"kubernetes.io/projected/9811d4e5-fe1b-4920-9601-fc298fb4ab66-kube-api-access-nfkcb\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.265598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.265587 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-dshm\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.265653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.265628 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tmp-dir\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.266207 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.266176 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tmp-dir\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.270893 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.266941 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-home\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.270893 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.267217 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-model-cache\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.284082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.278723 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tls-certs\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.284082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.281654 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-dshm\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.292500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.288818 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkcb\" (UniqueName: \"kubernetes.io/projected/9811d4e5-fe1b-4920-9601-fc298fb4ab66-kube-api-access-nfkcb\") pod \"sub-test-model-383fde-kserve-57697d44cf-csld4\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.391112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.390339 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:26:57.733368 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:26:57.733315 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9811d4e5_fe1b_4920_9601_fc298fb4ab66.slice/crio-a9ec1f8055a50371ce18155dd22561986f828c81b341ecce593cf93bd58c0876 WatchSource:0}: Error finding container a9ec1f8055a50371ce18155dd22561986f828c81b341ecce593cf93bd58c0876: Status 404 returned error can't find the container with id a9ec1f8055a50371ce18155dd22561986f828c81b341ecce593cf93bd58c0876 Nov 01 16:26:57.733583 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:57.733394 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4"] Nov 01 16:26:58.210452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:58.210409 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" event={"ID":"9811d4e5-fe1b-4920-9601-fc298fb4ab66","Type":"ContainerStarted","Data":"aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02"} Nov 01 16:26:58.210881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:58.210466 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" event={"ID":"9811d4e5-fe1b-4920-9601-fc298fb4ab66","Type":"ContainerStarted","Data":"a9ec1f8055a50371ce18155dd22561986f828c81b341ecce593cf93bd58c0876"} Nov 01 16:26:58.233571 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:58.233313 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" podStartSLOduration=1.233297565 podStartE2EDuration="1.233297565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:58.228650271 +0000 UTC m=+2358.373104915" watchObservedRunningTime="2026-11-01 16:26:58.233297565 +0000 UTC m=+2358.377752207" Nov 01 16:26:58.392284 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:26:58.392236 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:27:00.721016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:00.720972 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" podUID="1f4d856e-e307-4e7c-9149-e8d418fc599b" containerName="main" containerID="cri-o://f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29" gracePeriod=45 Nov 01 16:27:00.841947 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:00.841906 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" podUID="1f4d856e-e307-4e7c-9149-e8d418fc599b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.169:8000/ready\": dial tcp 10.132.0.169:8000: connect: connection refused" Nov 01 16:27:01.094215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.094185 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:27:01.175712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.175673 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4d856e-e307-4e7c-9149-e8d418fc599b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f4d856e-e307-4e7c-9149-e8d418fc599b-tls-certs\") pod \"1f4d856e-e307-4e7c-9149-e8d418fc599b\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " Nov 01 16:27:01.175890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.175802 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-home\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-home\") pod \"1f4d856e-e307-4e7c-9149-e8d418fc599b\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " Nov 01 16:27:01.175890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.175830 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-dshm\") pod \"1f4d856e-e307-4e7c-9149-e8d418fc599b\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " Nov 01 16:27:01.175994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.175897 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-tmp-dir\") pod \"1f4d856e-e307-4e7c-9149-e8d418fc599b\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " Nov 01 16:27:01.175994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.175936 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-model-cache\") pod \"1f4d856e-e307-4e7c-9149-e8d418fc599b\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " Nov 01 16:27:01.176158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.176084 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4d856e-e307-4e7c-9149-e8d418fc599b-kube-api-access-6gqk4\" (UniqueName: \"kubernetes.io/projected/1f4d856e-e307-4e7c-9149-e8d418fc599b-kube-api-access-6gqk4\") pod \"1f4d856e-e307-4e7c-9149-e8d418fc599b\" (UID: \"1f4d856e-e307-4e7c-9149-e8d418fc599b\") " Nov 01 16:27:01.176158 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.176120 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-home" pod "1f4d856e-e307-4e7c-9149-e8d418fc599b" (UID: "1f4d856e-e307-4e7c-9149-e8d418fc599b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:01.176276 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.176172 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-tmp-dir" pod "1f4d856e-e307-4e7c-9149-e8d418fc599b" (UID: "1f4d856e-e307-4e7c-9149-e8d418fc599b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:01.176336 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.176317 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-model-cache" pod "1f4d856e-e307-4e7c-9149-e8d418fc599b" (UID: "1f4d856e-e307-4e7c-9149-e8d418fc599b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:01.183097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.177648 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:01.183097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.177677 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:01.183097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.177692 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:01.183097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.178678 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-dshm" pod "1f4d856e-e307-4e7c-9149-e8d418fc599b" (UID: "1f4d856e-e307-4e7c-9149-e8d418fc599b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:01.183097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.178775 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4d856e-e307-4e7c-9149-e8d418fc599b-tls-certs" pod "1f4d856e-e307-4e7c-9149-e8d418fc599b" (UID: "1f4d856e-e307-4e7c-9149-e8d418fc599b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:01.183097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.178971 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4d856e-e307-4e7c-9149-e8d418fc599b-kube-api-access-6gqk4" pod "1f4d856e-e307-4e7c-9149-e8d418fc599b" (UID: "1f4d856e-e307-4e7c-9149-e8d418fc599b"). InnerVolumeSpecName "kube-api-access-6gqk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:01.262118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.262086 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" Nov 01 16:27:01.262118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.262104 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" event={"ID":"1f4d856e-e307-4e7c-9149-e8d418fc599b","Type":"ContainerDied","Data":"f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29"} Nov 01 16:27:01.262380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.262165 4911 scope.go:122] "RemoveContainer" containerID="f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29" Nov 01 16:27:01.262380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.262015 4911 generic.go:356] "Generic (PLEG): container finished" podID="1f4d856e-e307-4e7c-9149-e8d418fc599b" containerID="f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29" exitCode=0 Nov 01 16:27:01.262380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.262273 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp" event={"ID":"1f4d856e-e307-4e7c-9149-e8d418fc599b","Type":"ContainerDied","Data":"6ce9d4f99d8a852394de626c954eef22f9795d3e40ed9ee175b66c072ada8aeb"} Nov 01 16:27:01.278895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.278863 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f4d856e-e307-4e7c-9149-e8d418fc599b-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:01.278895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.278895 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f4d856e-e307-4e7c-9149-e8d418fc599b-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:01.279144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.278906 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqk4\" (UniqueName: \"kubernetes.io/projected/1f4d856e-e307-4e7c-9149-e8d418fc599b-kube-api-access-6gqk4\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:01.279317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.279213 4911 scope.go:122] "RemoveContainer" containerID="f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29" Nov 01 16:27:01.279754 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:27:01.279719 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29\": container with ID starting with f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29 not found: ID does not exist" containerID="f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29" Nov 01 16:27:01.279754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.279766 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29"} err="failed to get container status \"f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29\": rpc error: code = NotFound desc = could not find container \"f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29\": container with ID starting with f500d056877d6d40609330a5d3d6d243db0f8b7e8284f126628203223a836e29 not found: ID does not exist" Nov 01 16:27:01.294675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.294637 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp"] Nov 01 16:27:01.301659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.301625 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/auth-test-model-a9a967-kserve-7c978cdd97-qwbpp"] Nov 01 16:27:01.969278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:01.969227 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4d856e-e307-4e7c-9149-e8d418fc599b" path="/var/lib/kubelet/pods/1f4d856e-e307-4e7c-9149-e8d418fc599b/volumes" Nov 01 16:27:07.392701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:07.392660 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:27:07.405262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:07.405228 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:27:08.341982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:08.341951 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:27:11.013897 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:27:11.013855 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:27:16.227481 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.227444 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4"] Nov 01 16:27:16.228466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.228444 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1f4d856e-e307-4e7c-9149-e8d418fc599b" containerName="main" Nov 01 16:27:16.228546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.228468 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4d856e-e307-4e7c-9149-e8d418fc599b" containerName="main" Nov 01 16:27:16.228626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.228617 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4d856e-e307-4e7c-9149-e8d418fc599b" containerName="main" Nov 01 16:27:16.232790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.232763 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.247310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.247267 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"openshift-service-ca.crt\"" Nov 01 16:27:16.247677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.247650 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"kube-root-ca.crt\"" Nov 01 16:27:16.248722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.248695 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"default-dockercfg-7tn47\"" Nov 01 16:27:16.248859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.248787 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-a9a967\"/\"sub-test-model-a9a967-kserve-self-signed-certs\"" Nov 01 16:27:16.253193 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.253162 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4"] Nov 01 16:27:16.401331 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.401283 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-tmp-dir\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.401567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.401446 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcr5\" (UniqueName: \"kubernetes.io/projected/a8a96349-9c37-491b-b324-6bd81ed6f07d-kube-api-access-zmcr5\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.401567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.401522 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-dshm\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.401567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.401554 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a96349-9c37-491b-b324-6bd81ed6f07d-tls-certs\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.401747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.401689 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-model-cache\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.401747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.401719 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-home\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.505286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.505234 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-tmp-dir\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.505666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.505641 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcr5\" (UniqueName: \"kubernetes.io/projected/a8a96349-9c37-491b-b324-6bd81ed6f07d-kube-api-access-zmcr5\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.505836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.505819 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-dshm\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.505949 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.505934 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a96349-9c37-491b-b324-6bd81ed6f07d-tls-certs\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.506202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.506179 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-model-cache\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.506342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.506328 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-home\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.507052 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.507011 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-home\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.508131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.507438 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-tmp-dir\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.518442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.518413 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-model-cache\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.521836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.521807 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcr5\" (UniqueName: \"kubernetes.io/projected/a8a96349-9c37-491b-b324-6bd81ed6f07d-kube-api-access-zmcr5\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.560118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.559644 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-dshm\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.560118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.560117 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a96349-9c37-491b-b324-6bd81ed6f07d-tls-certs\") pod \"sub-test-model-a9a967-kserve-864487d8bc-qp9g4\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.573340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.573013 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:16.891462 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:27:16.891433 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8a96349_9c37_491b_b324_6bd81ed6f07d.slice/crio-6894e28fd2b4cf26443265ab507863dca55a8e313be3bac555c8266a588cac83 WatchSource:0}: Error finding container 6894e28fd2b4cf26443265ab507863dca55a8e313be3bac555c8266a588cac83: Status 404 returned error can't find the container with id 6894e28fd2b4cf26443265ab507863dca55a8e313be3bac555c8266a588cac83 Nov 01 16:27:16.893637 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:16.893562 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4"] Nov 01 16:27:17.567280 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:17.567241 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" event={"ID":"a8a96349-9c37-491b-b324-6bd81ed6f07d","Type":"ContainerStarted","Data":"7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79"} Nov 01 16:27:17.567789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:17.567293 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" event={"ID":"a8a96349-9c37-491b-b324-6bd81ed6f07d","Type":"ContainerStarted","Data":"6894e28fd2b4cf26443265ab507863dca55a8e313be3bac555c8266a588cac83"} Nov 01 16:27:17.575104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:17.575079 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:17.586033 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:17.585987 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" podStartSLOduration=1.585972795 podStartE2EDuration="1.585972795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:17.584693107 +0000 UTC m=+2377.729147749" watchObservedRunningTime="2026-11-01 16:27:17.585972795 +0000 UTC m=+2377.730427439" Nov 01 16:27:26.573952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:26.573863 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:26.587638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:26.587605 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:26.691115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:26.691011 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:27:32.975809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:32.975768 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt"] Nov 01 16:27:32.984445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:32.984416 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:32.992094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:32.992066 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt"] Nov 01 16:27:33.148092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148057 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148154 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148199 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148231 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmsx\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-kube-api-access-7bmsx\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148280 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20770135-5e84-4719-8058-95c73685f02a-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148389 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20770135-5e84-4719-8058-95c73685f02a-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148442 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148508 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.148570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.148545 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.249972 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.249936 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.249979 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250011 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250228 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250315 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250387 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250424 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250450 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmsx\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-kube-api-access-7bmsx\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250496 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20770135-5e84-4719-8058-95c73685f02a-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250558 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20770135-5e84-4719-8058-95c73685f02a-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250632 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250673 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.250716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.250709 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.251274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.251249 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20770135-5e84-4719-8058-95c73685f02a-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.252509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.252484 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.253060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.253027 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20770135-5e84-4719-8058-95c73685f02a-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.259018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.258992 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.259144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.259077 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmsx\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-kube-api-access-7bmsx\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.303771 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.303740 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:33.638241 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:27:33.638197 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20770135_5e84_4719_8058_95c73685f02a.slice/crio-b72c58753d6fbbf1f987343f9562cb6b69dd702ccdfcfa6f8cf0554c1c7a8e9c WatchSource:0}: Error finding container b72c58753d6fbbf1f987343f9562cb6b69dd702ccdfcfa6f8cf0554c1c7a8e9c: Status 404 returned error can't find the container with id b72c58753d6fbbf1f987343f9562cb6b69dd702ccdfcfa6f8cf0554c1c7a8e9c Nov 01 16:27:33.639202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.639171 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt"] Nov 01 16:27:33.641490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.641457 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:33.641590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.641523 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:33.641590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.641565 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:33.716409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:33.716366 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" event={"ID":"20770135-5e84-4719-8058-95c73685f02a","Type":"ContainerStarted","Data":"b72c58753d6fbbf1f987343f9562cb6b69dd702ccdfcfa6f8cf0554c1c7a8e9c"} Nov 01 16:27:34.726201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:34.726167 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" event={"ID":"20770135-5e84-4719-8058-95c73685f02a","Type":"ContainerStarted","Data":"253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce"} Nov 01 16:27:34.772066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:34.759959 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podStartSLOduration=2.759939381 podStartE2EDuration="2.759939381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:34.754272804 +0000 UTC m=+2394.898727446" watchObservedRunningTime="2026-11-01 16:27:34.759939381 +0000 UTC m=+2394.904394026" Nov 01 16:27:35.303863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:35.303822 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:35.305840 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:35.305809 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:27:35.305977 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:35.305861 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 16:27:36.304638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:36.304599 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:27:36.305053 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:36.304656 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 16:27:37.304107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:37.304064 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:27:37.304297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:37.304131 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 16:27:38.304103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:38.304061 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:27:38.304610 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:38.304139 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 16:27:39.304981 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:39.304939 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:27:39.305389 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:39.305006 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 16:27:40.312064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:40.308949 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 01 16:27:40.312064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:40.309008 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 01 16:27:40.703765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:40.703674 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:27:40.714055 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:40.714007 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:27:40.873585 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:27:40.873546 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:27:41.309645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:41.309609 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:41.310135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:41.310113 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:41.311054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:41.311016 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:27:49.323679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:27:49.323633 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt"] Nov 01 16:28:04.353843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.353774 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" podUID="4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" containerName="main" containerID="cri-o://0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648" gracePeriod=45 Nov 01 16:28:04.678447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.678413 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:28:04.755415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755339 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-model-cache\") pod \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " Nov 01 16:28:04.755624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755425 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-dshm\") pod \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " Nov 01 16:28:04.755624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755527 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tmp-dir\") pod \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " Nov 01 16:28:04.755624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755583 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-home\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-home\") pod \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " Nov 01 16:28:04.755807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755659 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-model-cache" pod "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" (UID: "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:04.755807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755694 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tls-certs\") pod \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " Nov 01 16:28:04.755807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755798 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-kube-api-access-p2tql\" (UniqueName: \"kubernetes.io/projected/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-kube-api-access-p2tql\") pod \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\" (UID: \"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e\") " Nov 01 16:28:04.755958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.755864 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tmp-dir" pod "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" (UID: "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:04.756082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.756020 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-home" pod "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" (UID: "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:04.756776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.756752 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:04.756925 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.756910 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:04.757052 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.757023 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:04.758526 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.758490 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tls-certs" pod "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" (UID: "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:04.758747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.758723 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-kube-api-access-p2tql" pod "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" (UID: "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e"). InnerVolumeSpecName "kube-api-access-p2tql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:04.766196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.766166 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-dshm" pod "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" (UID: "4c9c93d6-86b7-4cf7-b73f-46e5737dd48e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:04.859325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.859288 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:04.859325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.859318 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:04.859325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:04.859328 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2tql\" (UniqueName: \"kubernetes.io/projected/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e-kube-api-access-p2tql\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:05.054016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.053964 4911 generic.go:356] "Generic (PLEG): container finished" podID="4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" containerID="0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648" exitCode=0 Nov 01 16:28:05.054309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.054088 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" event={"ID":"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e","Type":"ContainerDied","Data":"0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648"} Nov 01 16:28:05.054309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.054127 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" event={"ID":"4c9c93d6-86b7-4cf7-b73f-46e5737dd48e","Type":"ContainerDied","Data":"a456fec77c980991ed67d0e056acf7fa6b94f4d15820141efb83554b324d6781"} Nov 01 16:28:05.054309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.054151 4911 scope.go:122] "RemoveContainer" containerID="0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648" Nov 01 16:28:05.054572 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.054366 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt" Nov 01 16:28:05.069582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.069554 4911 scope.go:122] "RemoveContainer" containerID="0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648" Nov 01 16:28:05.069886 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:28:05.069870 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648\": container with ID starting with 0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648 not found: ID does not exist" containerID="0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648" Nov 01 16:28:05.069944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.069895 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648"} err="failed to get container status \"0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648\": rpc error: code = NotFound desc = could not find container \"0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648\": container with ID starting with 0a450f1ff8b92555f0bdcf88b335e84efd9aa68cbd2828e03c5a5c88440b9648 not found: ID does not exist" Nov 01 16:28:05.085908 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.085875 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt"] Nov 01 16:28:05.090125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:05.090096 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c78897/auth-test-model-c78897-kserve-58d59fc4d5-9q9xt"] Nov 01 16:28:06.011884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:06.011842 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" path="/var/lib/kubelet/pods/4c9c93d6-86b7-4cf7-b73f-46e5737dd48e/volumes" Nov 01 16:28:08.100065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:08.099385 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.451147 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:08.447810 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.555077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:08.537576 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.683071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:08.681361 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.683071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:08.682369 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" podUID="a8a96349-9c37-491b-b324-6bd81ed6f07d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": context deadline exceeded" Nov 01 16:28:08.752304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:08.752259 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 01 16:28:09.102237 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.102125 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 01 16:28:09.103581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.103552 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:28:09.127620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.127577 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 01 16:28:09.130267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.127904 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.130267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.127938 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 01 16:28:09.135386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.135325 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.172:8000/ready\": context deadline exceeded" Nov 01 16:28:09.144440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.144136 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.160100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.156183 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 01 16:28:09.306064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.305449 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.372571 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.350170 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.427059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.425759 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 01 16:28:09.427059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.426152 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 01 16:28:09.448362 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.448319 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.448892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.448869 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:28:09.695954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.682225 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.695954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.687102 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:28:09.695954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.688231 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" podUID="a8a96349-9c37-491b-b324-6bd81ed6f07d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.181:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.695954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.688565 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:28:09.698966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.698925 4911 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.172:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.701265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.701240 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:28:09.767292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.754143 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:09.767292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.755340 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:28:09.802327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.798675 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:28:09.819295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.817700 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:28:09.898072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.888641 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:28:09.898975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:09.898944 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:28:10.871307 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:28:10.871268 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:28:12.132982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:12.132939 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt"] Nov 01 16:28:21.394127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:21.393757 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l"] Nov 01 16:28:21.394127 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:21.394008 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" containerID="cri-o://7f762e6ff816dd2c766d8c363312e6cc002c55c7f5fc94473f03c925956f5981" gracePeriod=30 Nov 01 16:28:21.401613 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:21.401576 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch"] Nov 01 16:28:21.401873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:21.401838 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" containerID="cri-o://d186c1b73b87c4cc72b884f91cf2ecc53443f19f63c298d275c2d37680c46ebe" gracePeriod=30 Nov 01 16:28:25.154492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:25.154454 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:28:25.154883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:25.154510 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:28:25.302522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:25.302483 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:28:25.302717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:25.302550 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:28:26.688547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.688222 4911 generic.go:356] "Generic (PLEG): container finished" podID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerID="d186c1b73b87c4cc72b884f91cf2ecc53443f19f63c298d275c2d37680c46ebe" exitCode=0 Nov 01 16:28:26.688547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.688385 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" event={"ID":"9192cc03-71c8-4c48-a288-f7d4caefd426","Type":"ContainerDied","Data":"d186c1b73b87c4cc72b884f91cf2ecc53443f19f63c298d275c2d37680c46ebe"} Nov 01 16:28:26.692272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.691357 4911 generic.go:356] "Generic (PLEG): container finished" podID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerID="7f762e6ff816dd2c766d8c363312e6cc002c55c7f5fc94473f03c925956f5981" exitCode=0 Nov 01 16:28:26.693097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.693030 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" event={"ID":"21c3e2b6-d2d0-428f-907a-aa53f210f559","Type":"ContainerDied","Data":"7f762e6ff816dd2c766d8c363312e6cc002c55c7f5fc94473f03c925956f5981"} Nov 01 16:28:26.802809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.802777 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:28:26.842968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.842934 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:28:26.967197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967094 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-certs\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.967197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967152 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-token\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-token\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.967197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967187 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-certs\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967232 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-podinfo\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967284 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-envoy\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.967507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967309 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-socket\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.967507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967370 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-credential-socket\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967428 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-data\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967458 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-token\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-token\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967465 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-certs" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.967926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-kube-api-access-vrkqs\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-kube-api-access-vrkqs\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967624 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-kube-api-access-z49vk\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-kube-api-access-z49vk\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.967926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967645 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-credential-socket" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.967926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967711 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-envoy\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967774 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c3e2b6-d2d0-428f-907a-aa53f210f559-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c3e2b6-d2d0-428f-907a-aa53f210f559-istiod-ca-cert\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.967926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967788 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-socket" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.967926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967879 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-podinfo\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.968323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967959 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9192cc03-71c8-4c48-a288-f7d4caefd426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9192cc03-71c8-4c48-a288-f7d4caefd426-istiod-ca-cert\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.968323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.967987 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-socket\") pod \"21c3e2b6-d2d0-428f-907a-aa53f210f559\" (UID: \"21c3e2b6-d2d0-428f-907a-aa53f210f559\") " Nov 01 16:28:26.968323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.968019 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c3e2b6-d2d0-428f-907a-aa53f210f559-istiod-ca-cert" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:26.968323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.968019 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-data\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.968323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.968131 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-credential-socket\") pod \"9192cc03-71c8-4c48-a288-f7d4caefd426\" (UID: \"9192cc03-71c8-4c48-a288-f7d4caefd426\") " Nov 01 16:28:26.968323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.968182 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-data" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.968664 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.968383 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-certs" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.968747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.968712 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9192cc03-71c8-4c48-a288-f7d4caefd426-istiod-ca-cert" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:26.969231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969208 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-credential-socket" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.969231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969211 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-socket" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.969436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969411 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-data" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.969648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969628 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.969736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969654 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c3e2b6-d2d0-428f-907a-aa53f210f559-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.969736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969669 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9192cc03-71c8-4c48-a288-f7d4caefd426-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.969736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969684 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.969736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969697 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.969736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969711 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.969736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.969726 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.970807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.970782 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-podinfo" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:26.971666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.971631 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-envoy" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.973406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.973174 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-kube-api-access-vrkqs" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "kube-api-access-vrkqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:26.973750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.973720 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-token" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:26.973865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.973758 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-kube-api-access-z49vk" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "kube-api-access-z49vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:26.977374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.977348 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-podinfo" pod "21c3e2b6-d2d0-428f-907a-aa53f210f559" (UID: "21c3e2b6-d2d0-428f-907a-aa53f210f559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:26.977466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.977375 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-envoy" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.978576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:26.978553 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-token" pod "9192cc03-71c8-4c48-a288-f7d4caefd426" (UID: "9192cc03-71c8-4c48-a288-f7d4caefd426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:27.071813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071769 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrkqs\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-kube-api-access-vrkqs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.071813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071805 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z49vk\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-kube-api-access-z49vk\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.071813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071817 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071828 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071837 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071846 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071855 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071864 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071872 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9192cc03-71c8-4c48-a288-f7d4caefd426-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071881 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.072062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.071889 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c3e2b6-d2d0-428f-907a-aa53f210f559-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.162074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.161964 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerName="main" containerID="cri-o://0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f" gracePeriod=45 Nov 01 16:28:27.469673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.469639 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:28:27.582947 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.582852 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-home\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-home\") pod \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " Nov 01 16:28:27.583171 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.582978 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tls-certs\") pod \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " Nov 01 16:28:27.583171 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.583014 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-dshm\") pod \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " Nov 01 16:28:27.583831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.583169 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-kube-api-access-qtdvj\" (UniqueName: \"kubernetes.io/projected/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-kube-api-access-qtdvj\") pod \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " Nov 01 16:28:27.583831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.583202 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-home" pod "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" (UID: "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.583831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.583373 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tmp-dir\") pod \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " Nov 01 16:28:27.583831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.583483 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-model-cache\") pod \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\" (UID: \"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0\") " Nov 01 16:28:27.583831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.583616 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tmp-dir" pod "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" (UID: "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.583831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.583762 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-model-cache" pod "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" (UID: "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.584422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.584397 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.584473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.584427 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.584473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.584442 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.585567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.585541 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-dshm" pod "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" (UID: "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.585848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.585829 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tls-certs" pod "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" (UID: "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:27.585926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.585873 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-kube-api-access-qtdvj" pod "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" (UID: "1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0"). InnerVolumeSpecName "kube-api-access-qtdvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:27.692823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.692685 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.692823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.692726 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.692823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.692745 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtdvj\" (UniqueName: \"kubernetes.io/projected/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0-kube-api-access-qtdvj\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.718873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.718836 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" event={"ID":"9192cc03-71c8-4c48-a288-f7d4caefd426","Type":"ContainerDied","Data":"c532efebbecb243806f045f1f912985f669179dd25e2f2bda96833da39178b9b"} Nov 01 16:28:27.719115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.718936 4911 scope.go:122] "RemoveContainer" containerID="d186c1b73b87c4cc72b884f91cf2ecc53443f19f63c298d275c2d37680c46ebe" Nov 01 16:28:27.719209 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.719152 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch" Nov 01 16:28:27.721956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.721922 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" event={"ID":"21c3e2b6-d2d0-428f-907a-aa53f210f559","Type":"ContainerDied","Data":"2febe53a16c99361fd3b8701091f27bea46303b4e64cd6bb0201c077120f865b"} Nov 01 16:28:27.722260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.722052 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l" Nov 01 16:28:27.725399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.725369 4911 generic.go:356] "Generic (PLEG): container finished" podID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerID="0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f" exitCode=0 Nov 01 16:28:27.732086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.725573 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" event={"ID":"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0","Type":"ContainerDied","Data":"0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f"} Nov 01 16:28:27.732086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.725777 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" event={"ID":"1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0","Type":"ContainerDied","Data":"5766c0a7cdb0bdf7b45e5ade2601f9a3c6b2149c0c93e653ed9e96c76e3cc72d"} Nov 01 16:28:27.732086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.725853 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt" Nov 01 16:28:27.744857 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.744832 4911 scope.go:122] "RemoveContainer" containerID="7f762e6ff816dd2c766d8c363312e6cc002c55c7f5fc94473f03c925956f5981" Nov 01 16:28:27.754382 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.754356 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch"] Nov 01 16:28:27.757285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.757262 4911 scope.go:122] "RemoveContainer" containerID="0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f" Nov 01 16:28:27.759550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.759516 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-mnzch"] Nov 01 16:28:27.767192 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.767172 4911 scope.go:122] "RemoveContainer" containerID="0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f" Nov 01 16:28:27.767494 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:28:27.767474 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f\": container with ID starting with 0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f not found: ID does not exist" containerID="0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f" Nov 01 16:28:27.767553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.767503 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f"} err="failed to get container status \"0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f\": rpc error: code = NotFound desc = could not find container \"0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f\": container with ID starting with 0368fb39f3f7a110ec69ee6b29846055ae2f3a7780e8b13d966e44d746573c8f not found: ID does not exist" Nov 01 16:28:27.769970 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.769942 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt"] Nov 01 16:28:27.777092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.777064 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/sub-test-model-f01de5-kserve-6b4d5fc767-9hljt"] Nov 01 16:28:27.788947 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.788920 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l"] Nov 01 16:28:27.792801 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.792775 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-qlx2l"] Nov 01 16:28:27.968829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.968740 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" path="/var/lib/kubelet/pods/1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0/volumes" Nov 01 16:28:27.969587 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.969567 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" path="/var/lib/kubelet/pods/21c3e2b6-d2d0-428f-907a-aa53f210f559/volumes" Nov 01 16:28:27.970233 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:27.970217 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" path="/var/lib/kubelet/pods/9192cc03-71c8-4c48-a288-f7d4caefd426/volumes" Nov 01 16:28:35.488857 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.488818 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh"] Nov 01 16:28:35.489546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489523 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" Nov 01 16:28:35.489546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489543 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" Nov 01 16:28:35.489745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489581 4911 cpu_manager.go:414] "containerMap: removing container" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" Nov 01 16:28:35.489745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489587 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" Nov 01 16:28:35.489745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489601 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" containerName="main" Nov 01 16:28:35.489745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489606 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" containerName="main" Nov 01 16:28:35.489745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489619 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerName="main" Nov 01 16:28:35.489745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489624 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerName="main" Nov 01 16:28:35.490097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489765 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c3e2b6-d2d0-428f-907a-aa53f210f559" containerName="istio-proxy" Nov 01 16:28:35.490097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489776 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9c93d6-86b7-4cf7-b73f-46e5737dd48e" containerName="main" Nov 01 16:28:35.490097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489784 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edffe25-b7c8-4f4e-b7fd-84ac4879cfe0" containerName="main" Nov 01 16:28:35.490097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.489792 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9192cc03-71c8-4c48-a288-f7d4caefd426" containerName="istio-proxy" Nov 01 16:28:35.495564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.495533 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.500639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.500610 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh"] Nov 01 16:28:35.509565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.509540 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"kube-root-ca.crt\"" Nov 01 16:28:35.509766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.509578 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"default-dockercfg-mjz9s\"" Nov 01 16:28:35.509940 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.509647 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"openshift-service-ca.crt\"" Nov 01 16:28:35.510089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.509705 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-f01de5\"/\"rate-test-model-f01de5-kserve-self-signed-certs\"" Nov 01 16:28:35.557671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.557602 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-model-cache\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.558095 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.558066 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tmp-dir\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.558319 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.558298 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslg4\" (UniqueName: \"kubernetes.io/projected/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-kube-api-access-mslg4\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.558545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.558523 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-home\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.558687 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.558669 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-dshm\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.558814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.558798 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tls-certs\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.662150 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.662105 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-home\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.674264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.674225 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-dshm\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.674567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.674539 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tls-certs\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.689125 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.674826 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-model-cache\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.689964 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.689926 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tmp-dir\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.690243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.690222 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslg4\" (UniqueName: \"kubernetes.io/projected/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-kube-api-access-mslg4\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.697112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.679436 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-model-cache\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.697288 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.662843 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-home\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.697401 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.691448 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tmp-dir\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.697505 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.692155 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tls-certs\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.718135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.714724 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-dshm\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.738072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.732090 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslg4\" (UniqueName: \"kubernetes.io/projected/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-kube-api-access-mslg4\") pod \"rate-test-model-f01de5-kserve-6bb95857df-8sdhh\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:35.828108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:35.824760 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:36.156488 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:28:36.156456 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc50ef3d7_ab6e_4009_8a8d_78d80de3ead5.slice/crio-a97733fecbdb322e76209cbc5fe42efbf4877cb27e59fa8f72bbb89246b0221f WatchSource:0}: Error finding container a97733fecbdb322e76209cbc5fe42efbf4877cb27e59fa8f72bbb89246b0221f: Status 404 returned error can't find the container with id a97733fecbdb322e76209cbc5fe42efbf4877cb27e59fa8f72bbb89246b0221f Nov 01 16:28:36.156812 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:36.156784 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh"] Nov 01 16:28:36.857380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:36.857339 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" event={"ID":"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5","Type":"ContainerStarted","Data":"8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0"} Nov 01 16:28:36.857810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:36.857391 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" event={"ID":"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5","Type":"ContainerStarted","Data":"a97733fecbdb322e76209cbc5fe42efbf4877cb27e59fa8f72bbb89246b0221f"} Nov 01 16:28:36.877085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:36.876989 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" podStartSLOduration=1.876973149 podStartE2EDuration="1.876973149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:36.876287016 +0000 UTC m=+2457.020741657" watchObservedRunningTime="2026-11-01 16:28:36.876973149 +0000 UTC m=+2457.021427789" Nov 01 16:28:37.268469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.268431 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m"] Nov 01 16:28:37.273006 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.272987 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.278773 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.278749 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-734859-openshift-default-dockercfg-gwnw2\"" Nov 01 16:28:37.294324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.294299 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m"] Nov 01 16:28:37.428776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.428736 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.428937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.428846 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.428937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.428910 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.429137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.429102 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.429237 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.429171 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.429327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.429305 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.429376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.429352 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.429414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.429387 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.429470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.429412 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bmc\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-kube-api-access-w5bmc\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.530980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.530875 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.530980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.530934 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.530980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.530966 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bmc\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-kube-api-access-w5bmc\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.531303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.531108 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.531303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.531175 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.531303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.531260 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.531469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.531338 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.531469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.531384 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.531596 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.531513 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.532055 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.532013 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.532340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.532313 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.532595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.532573 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.533803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.533777 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.535069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.535026 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.537011 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.536981 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.537279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.537255 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.544216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.544183 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bmc\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-kube-api-access-w5bmc\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.544693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.544663 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.588993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.588942 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:37.830697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.830612 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:37.954958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.954922 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m"] Nov 01 16:28:37.965635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.965601 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:37.965746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.965688 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:37.965746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:37.965723 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:38.766066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.749304 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4"] Nov 01 16:28:38.787099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.787013 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4"] Nov 01 16:28:38.787299 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.787218 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.792190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.792164 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-383fde-openshift-default-dockercfg-lxnqs\"" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860138 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqn6t\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-kube-api-access-kqn6t\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860208 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9a8165-4040-46b2-b600-e423d00f851c-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860266 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860303 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860338 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9a8165-4040-46b2-b600-e423d00f851c-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860385 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860462 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860522 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.872064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.860727 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.935413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.935376 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" event={"ID":"b4ecd3e2-ea3f-461e-b436-36910fc14f95","Type":"ContainerStarted","Data":"b348296c028eb62561e4c632dd256474ccb931b0aa03068645645e676888fff6"} Nov 01 16:28:38.935621 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.935452 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" event={"ID":"b4ecd3e2-ea3f-461e-b436-36910fc14f95","Type":"ContainerStarted","Data":"afc8ba2bb46a0c8484aa64e6c0dd937851cea205d850eb92c050f8ee41550e72"} Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975190 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975248 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9a8165-4040-46b2-b600-e423d00f851c-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975291 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975357 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975412 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975594 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975714 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqn6t\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-kube-api-access-kqn6t\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975764 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9a8165-4040-46b2-b600-e423d00f851c-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.976062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.975819 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.977007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.976332 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.978060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.977629 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podStartSLOduration=1.977612333 podStartE2EDuration="1.977612333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:38.9756658 +0000 UTC m=+2459.120120442" watchObservedRunningTime="2026-11-01 16:28:38.977612333 +0000 UTC m=+2459.122066975" Nov 01 16:28:38.980101 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.979283 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.980101 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.979904 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9a8165-4040-46b2-b600-e423d00f851c-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.980265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.980167 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.980265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.980248 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.982923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.982891 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.985913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.985861 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9a8165-4040-46b2-b600-e423d00f851c-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.988075 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.988010 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqn6t\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-kube-api-access-kqn6t\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:38.988548 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:38.988520 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:39.103839 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.103759 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:39.419167 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:28:39.419132 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9a8165_4040_46b2_b600_e423d00f851c.slice/crio-24609746c9b4b1cbecbbde7c8ef892b347408044c0b07751669efd8bad0e8954 WatchSource:0}: Error finding container 24609746c9b4b1cbecbbde7c8ef892b347408044c0b07751669efd8bad0e8954: Status 404 returned error can't find the container with id 24609746c9b4b1cbecbbde7c8ef892b347408044c0b07751669efd8bad0e8954 Nov 01 16:28:39.421091 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.421064 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4"] Nov 01 16:28:39.422243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.422208 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:39.422324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.422292 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:39.422377 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.422340 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:39.600141 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.593407 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:39.620072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.616205 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 01 16:28:39.620072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.616260 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 01 16:28:39.946784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.946727 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" event={"ID":"5b9a8165-4040-46b2-b600-e423d00f851c","Type":"ContainerStarted","Data":"5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e"} Nov 01 16:28:39.946995 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:39.946809 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" event={"ID":"5b9a8165-4040-46b2-b600-e423d00f851c","Type":"ContainerStarted","Data":"24609746c9b4b1cbecbbde7c8ef892b347408044c0b07751669efd8bad0e8954"} Nov 01 16:28:40.106174 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:40.104654 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:40.109063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:40.108904 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:28:40.109063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:40.108973 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 16:28:40.598128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:40.592270 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 01 16:28:40.598128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:40.592346 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 01 16:28:41.110285 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:28:41.110244 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:28:41.114943 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:41.114427 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:28:41.114943 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:41.114489 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 16:28:41.607224 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:41.607182 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 01 16:28:41.607443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:41.607246 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 01 16:28:42.105545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:42.105090 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:28:42.105545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:42.105155 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 16:28:42.596116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:42.595646 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 01 16:28:42.596116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:42.595714 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 01 16:28:43.108102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:43.104733 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:28:43.108102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:43.104798 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 16:28:43.603066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:43.597402 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 01 16:28:43.603066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:43.597483 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 01 16:28:44.110710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:44.107621 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:28:44.110710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:44.107682 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 16:28:44.590451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:44.590410 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 01 16:28:44.590661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:44.590494 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 01 16:28:45.104623 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.104588 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:28:45.105076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.104727 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 16:28:45.595073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.595013 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:45.595580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.595552 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:45.596328 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.596308 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:28:45.629791 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.626646 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podStartSLOduration=7.626624469 podStartE2EDuration="7.626624469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:39.980642042 +0000 UTC m=+2460.125096684" watchObservedRunningTime="2026-11-01 16:28:45.626624469 +0000 UTC m=+2465.771079111" Nov 01 16:28:45.830737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.830706 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:45.844371 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:45.844336 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:46.104335 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:46.104297 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:28:46.104501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:46.104351 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 01 16:28:46.167194 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:46.167156 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:28:47.109423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:47.109394 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:47.109899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:47.109868 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:47.110720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:47.110699 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:28:48.249540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:48.249499 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h"] Nov 01 16:28:48.250294 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:48.250240 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" containerID="cri-o://4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229" gracePeriod=30 Nov 01 16:28:48.279409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:48.279368 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr"] Nov 01 16:28:48.280191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:48.279911 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" containerID="cri-o://73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68" gracePeriod=30 Nov 01 16:28:48.288815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:48.288233 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt"] Nov 01 16:28:48.288815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:48.288476 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" containerID="cri-o://253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce" gracePeriod=30 Nov 01 16:28:53.624419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.624380 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:28:53.674994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.674962 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:28:53.708378 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.708337 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6"] Nov 01 16:28:53.737387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737344 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-certs\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.737580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737418 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5061671b-c676-4375-b9c5-53637897fb5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5061671b-c676-4375-b9c5-53637897fb5f-istio-podinfo\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.737580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737492 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-data\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.737701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-socket\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.737701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737685 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-envoy\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.737808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737751 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5061671b-c676-4375-b9c5-53637897fb5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5061671b-c676-4375-b9c5-53637897fb5f-istiod-ca-cert\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.737862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737828 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-credential-socket\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.740584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737932 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-istio-token\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-istio-token\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.740584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.737967 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-kube-api-access-2thm9\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-kube-api-access-2thm9\") pod \"5061671b-c676-4375-b9c5-53637897fb5f\" (UID: \"5061671b-c676-4375-b9c5-53637897fb5f\") " Nov 01 16:28:53.740958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.740915 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-socket" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.741262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.741232 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-certs" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.743361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.743333 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-credential-socket" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.743543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.743517 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-data" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.744635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.744602 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5061671b-c676-4375-b9c5-53637897fb5f-istiod-ca-cert" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:53.747365 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.746582 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-istio-token" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.748794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.748025 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-envoy" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.758109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.758053 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5061671b-c676-4375-b9c5-53637897fb5f-istio-podinfo" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:53.759642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.759617 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:28:53.759971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.759934 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs"] Nov 01 16:28:53.761645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.761617 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-kube-api-access-2thm9" pod "5061671b-c676-4375-b9c5-53637897fb5f" (UID: "5061671b-c676-4375-b9c5-53637897fb5f"). InnerVolumeSpecName "kube-api-access-2thm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.764591 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.764559 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl"] Nov 01 16:28:53.839561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839456 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-kube-api-access-7bmsx\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-kube-api-access-7bmsx\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.839561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839494 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-istio-token\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-istio-token\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.839561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839528 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-credential-socket\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.839874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839615 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-kube-api-access-mhd2x\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-kube-api-access-mhd2x\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.839874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839645 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-certs\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.839874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839706 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20770135-5e84-4719-8058-95c73685f02a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20770135-5e84-4719-8058-95c73685f02a-istio-podinfo\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.839874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839763 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-socket\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.839874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839796 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20770135-5e84-4719-8058-95c73685f02a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20770135-5e84-4719-8058-95c73685f02a-istiod-ca-cert\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.839874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839827 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0105a45c-db4b-4929-950e-4714d7ac9369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0105a45c-db4b-4929-950e-4714d7ac9369-istio-podinfo\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.839874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839867 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-data\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839880 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-credential-socket" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839912 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-socket\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.839997 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-certs\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840032 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-data\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840125 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-envoy\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840147 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-socket" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840173 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-credential-socket\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840218 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-envoy\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840250 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0105a45c-db4b-4929-950e-4714d7ac9369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0105a45c-db4b-4929-950e-4714d7ac9369-istiod-ca-cert\") pod \"0105a45c-db4b-4929-950e-4714d7ac9369\" (UID: \"0105a45c-db4b-4929-950e-4714d7ac9369\") " Nov 01 16:28:53.840297 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840284 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-istio-token\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-istio-token\") pod \"20770135-5e84-4719-8058-95c73685f02a\" (UID: \"20770135-5e84-4719-8058-95c73685f02a\") " Nov 01 16:28:53.840858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840306 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-data" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.840858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840519 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-certs" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.840858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.840793 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20770135-5e84-4719-8058-95c73685f02a-istiod-ca-cert" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:53.841213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841182 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841216 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841232 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thm9\" (UniqueName: \"kubernetes.io/projected/5061671b-c676-4375-b9c5-53637897fb5f-kube-api-access-2thm9\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841248 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841262 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841277 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5061671b-c676-4375-b9c5-53637897fb5f-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841292 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841306 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841321 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.841340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841334 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.842376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841348 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20770135-5e84-4719-8058-95c73685f02a-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.842376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841362 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.842376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841376 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5061671b-c676-4375-b9c5-53637897fb5f-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.842376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841390 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5061671b-c676-4375-b9c5-53637897fb5f-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.842376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.841852 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-certs" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.842376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.842055 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-socket" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.842376 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.842229 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-data" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.843245 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.843220 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0105a45c-db4b-4929-950e-4714d7ac9369-istiod-ca-cert" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:53.843342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.843253 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-credential-socket" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.844120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.844091 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-envoy" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.845498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.845472 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-istio-token" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.845498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.845477 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0105a45c-db4b-4929-950e-4714d7ac9369-istio-podinfo" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:53.845685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.845524 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-kube-api-access-mhd2x" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "kube-api-access-mhd2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.845685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.845659 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-istio-token" pod "0105a45c-db4b-4929-950e-4714d7ac9369" (UID: "0105a45c-db4b-4929-950e-4714d7ac9369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.845685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.845669 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-kube-api-access-7bmsx" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "kube-api-access-7bmsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.846605 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.846583 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20770135-5e84-4719-8058-95c73685f02a-istio-podinfo" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:53.847476 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:53.847453 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-envoy" pod "20770135-5e84-4719-8058-95c73685f02a" (UID: "20770135-5e84-4719-8058-95c73685f02a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086749 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0105a45c-db4b-4929-950e-4714d7ac9369-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086793 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086807 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086823 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086836 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086860 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20770135-5e84-4719-8058-95c73685f02a-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086874 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0105a45c-db4b-4929-950e-4714d7ac9369-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086886 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0105a45c-db4b-4929-950e-4714d7ac9369-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086903 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086916 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bmsx\" (UniqueName: \"kubernetes.io/projected/20770135-5e84-4719-8058-95c73685f02a-kube-api-access-7bmsx\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086942 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086955 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhd2x\" (UniqueName: \"kubernetes.io/projected/0105a45c-db4b-4929-950e-4714d7ac9369-kube-api-access-mhd2x\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.096113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.086968 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20770135-5e84-4719-8058-95c73685f02a-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:54.357825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.357730 4911 generic.go:356] "Generic (PLEG): container finished" podID="20770135-5e84-4719-8058-95c73685f02a" containerID="253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce" exitCode=0 Nov 01 16:28:54.357825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.357783 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" Nov 01 16:28:54.358116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.357813 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" event={"ID":"20770135-5e84-4719-8058-95c73685f02a","Type":"ContainerDied","Data":"253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce"} Nov 01 16:28:54.358116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.357880 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt" event={"ID":"20770135-5e84-4719-8058-95c73685f02a","Type":"ContainerDied","Data":"b72c58753d6fbbf1f987343f9562cb6b69dd702ccdfcfa6f8cf0554c1c7a8e9c"} Nov 01 16:28:54.358116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.357909 4911 scope.go:122] "RemoveContainer" containerID="253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce" Nov 01 16:28:54.360395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.360362 4911 generic.go:356] "Generic (PLEG): container finished" podID="0105a45c-db4b-4929-950e-4714d7ac9369" containerID="73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68" exitCode=0 Nov 01 16:28:54.360550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.360457 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" Nov 01 16:28:54.360550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.360502 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" event={"ID":"0105a45c-db4b-4929-950e-4714d7ac9369","Type":"ContainerDied","Data":"73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68"} Nov 01 16:28:54.360550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.360544 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr" event={"ID":"0105a45c-db4b-4929-950e-4714d7ac9369","Type":"ContainerDied","Data":"c44d34e59d96cfe7398c42583bc5c6b4b13f62d7852abaed9c62359fb72f6528"} Nov 01 16:28:54.368086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.365575 4911 generic.go:356] "Generic (PLEG): container finished" podID="5061671b-c676-4375-b9c5-53637897fb5f" containerID="4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229" exitCode=0 Nov 01 16:28:54.368086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.365654 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" event={"ID":"5061671b-c676-4375-b9c5-53637897fb5f","Type":"ContainerDied","Data":"4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229"} Nov 01 16:28:54.368086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.365685 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" Nov 01 16:28:54.368086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.365690 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h" event={"ID":"5061671b-c676-4375-b9c5-53637897fb5f","Type":"ContainerDied","Data":"a51a96306e1f5ebbbedb670c2998ece3c48a072eddfde35d47f3ea37eba9474a"} Nov 01 16:28:54.374929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.372328 4911 scope.go:122] "RemoveContainer" containerID="253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce" Nov 01 16:28:54.374929 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:28:54.372614 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce\": container with ID starting with 253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce not found: ID does not exist" containerID="253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce" Nov 01 16:28:54.374929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.372637 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce"} err="failed to get container status \"253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce\": rpc error: code = NotFound desc = could not find container \"253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce\": container with ID starting with 253db66ae08cadc08d0c881a010334d2d21bc1da4ab877583a321b421ead9dce not found: ID does not exist" Nov 01 16:28:54.374929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.372679 4911 scope.go:122] "RemoveContainer" containerID="73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68" Nov 01 16:28:54.386858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.386811 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr"] Nov 01 16:28:54.396116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.395547 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-gxmtr"] Nov 01 16:28:54.405783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.403568 4911 scope.go:122] "RemoveContainer" containerID="73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68" Nov 01 16:28:54.405783 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:28:54.404198 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68\": container with ID starting with 73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68 not found: ID does not exist" containerID="73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68" Nov 01 16:28:54.405783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.404274 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68"} err="failed to get container status \"73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68\": rpc error: code = NotFound desc = could not find container \"73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68\": container with ID starting with 73c7eaedeeae51affa8e0a41dfd3a1b582e9632d49cb76ad10b2bdbd22fcff68 not found: ID does not exist" Nov 01 16:28:54.405783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.404328 4911 scope.go:122] "RemoveContainer" containerID="4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229" Nov 01 16:28:54.493672 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.493627 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt"] Nov 01 16:28:54.493954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.493703 4911 scope.go:122] "RemoveContainer" containerID="4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229" Nov 01 16:28:54.495248 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:28:54.495216 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229\": container with ID starting with 4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229 not found: ID does not exist" containerID="4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229" Nov 01 16:28:54.495577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.495473 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229"} err="failed to get container status \"4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229\": rpc error: code = NotFound desc = could not find container \"4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229\": container with ID starting with 4962f9f82aa8daad558b95527c06f46838accfa4fac6810bcc703fa54f384229 not found: ID does not exist" Nov 01 16:28:54.495712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.495666 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-78zkt"] Nov 01 16:28:54.511856 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.511825 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h"] Nov 01 16:28:54.517238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:54.517207 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-rc28h"] Nov 01 16:28:55.970466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:55.970431 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" path="/var/lib/kubelet/pods/0105a45c-db4b-4929-950e-4714d7ac9369/volumes" Nov 01 16:28:55.970992 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:55.970974 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20770135-5e84-4719-8058-95c73685f02a" path="/var/lib/kubelet/pods/20770135-5e84-4719-8058-95c73685f02a/volumes" Nov 01 16:28:55.971577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:28:55.971557 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5061671b-c676-4375-b9c5-53637897fb5f" path="/var/lib/kubelet/pods/5061671b-c676-4375-b9c5-53637897fb5f/volumes" Nov 01 16:29:08.799105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:08.798412 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" containerID="cri-o://f761c8391753ff28c3edd55997bee44482d57c2d3862a3c574768e316b09e87f" gracePeriod=45 Nov 01 16:29:08.907064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:08.902426 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" containerID="cri-o://380bcd066cb147c98ca38e4a86fcaa1c93cf6579d36ea78d9f87c34e679246f3" gracePeriod=45 Nov 01 16:29:08.907064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:08.906161 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" containerID="cri-o://bfae6e904d4f5b826f34f3258993dc346f115b98628346a5aaa3acea6582b6fb" gracePeriod=45 Nov 01 16:29:09.180058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.178433 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused" Nov 01 16:29:09.373258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.373120 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": dial tcp 10.132.0.86:8000: connect: connection refused" Nov 01 16:29:09.768533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.768205 4911 generic.go:356] "Generic (PLEG): container finished" podID="e441066e-613b-4f47-be65-0e60deb73fc2" containerID="380bcd066cb147c98ca38e4a86fcaa1c93cf6579d36ea78d9f87c34e679246f3" exitCode=0 Nov 01 16:29:09.768533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.768478 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" event={"ID":"e441066e-613b-4f47-be65-0e60deb73fc2","Type":"ContainerDied","Data":"380bcd066cb147c98ca38e4a86fcaa1c93cf6579d36ea78d9f87c34e679246f3"} Nov 01 16:29:09.772976 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.772680 4911 generic.go:356] "Generic (PLEG): container finished" podID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerID="bfae6e904d4f5b826f34f3258993dc346f115b98628346a5aaa3acea6582b6fb" exitCode=0 Nov 01 16:29:09.772976 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.772910 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" event={"ID":"686a5cc9-26a2-44e2-8a1a-ba5001f492b7","Type":"ContainerDied","Data":"bfae6e904d4f5b826f34f3258993dc346f115b98628346a5aaa3acea6582b6fb"} Nov 01 16:29:09.780472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.780373 4911 generic.go:356] "Generic (PLEG): container finished" podID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerID="f761c8391753ff28c3edd55997bee44482d57c2d3862a3c574768e316b09e87f" exitCode=0 Nov 01 16:29:09.780472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.780418 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" event={"ID":"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6","Type":"ContainerDied","Data":"f761c8391753ff28c3edd55997bee44482d57c2d3862a3c574768e316b09e87f"} Nov 01 16:29:09.788309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.787994 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:29:09.795161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.792363 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:29:09.798672 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.798647 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.890808 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-home\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-home\") pod \"e441066e-613b-4f47-be65-0e60deb73fc2\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.890891 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tmp-dir\") pod \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.890975 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-model-cache\") pod \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891001 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e441066e-613b-4f47-be65-0e60deb73fc2-kube-api-access-ks65v\" (UniqueName: \"kubernetes.io/projected/e441066e-613b-4f47-be65-0e60deb73fc2-kube-api-access-ks65v\") pod \"e441066e-613b-4f47-be65-0e60deb73fc2\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891081 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-kube-api-access-89bsx\" (UniqueName: \"kubernetes.io/projected/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-kube-api-access-89bsx\") pod \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891098 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tls-certs\") pod \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891120 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-home\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-home\") pod \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891166 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tmp-dir\") pod \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891186 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tls-certs\") pod \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891206 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-home\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-home\") pod \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891230 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-model-cache\") pod \"e441066e-613b-4f47-be65-0e60deb73fc2\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891313 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-dshm\") pod \"e441066e-613b-4f47-be65-0e60deb73fc2\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891340 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e441066e-613b-4f47-be65-0e60deb73fc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e441066e-613b-4f47-be65-0e60deb73fc2-tls-certs\") pod \"e441066e-613b-4f47-be65-0e60deb73fc2\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891362 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-tmp-dir\") pod \"e441066e-613b-4f47-be65-0e60deb73fc2\" (UID: \"e441066e-613b-4f47-be65-0e60deb73fc2\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891384 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-model-cache\") pod \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891405 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-kube-api-access-xjmbr\" (UniqueName: \"kubernetes.io/projected/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-kube-api-access-xjmbr\") pod \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891442 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-dshm\") pod \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\" (UID: \"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.891466 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-dshm\") pod \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\" (UID: \"686a5cc9-26a2-44e2-8a1a-ba5001f492b7\") " Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.892945 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-model-cache" pod "e441066e-613b-4f47-be65-0e60deb73fc2" (UID: "e441066e-613b-4f47-be65-0e60deb73fc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.893183 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-home" pod "686a5cc9-26a2-44e2-8a1a-ba5001f492b7" (UID: "686a5cc9-26a2-44e2-8a1a-ba5001f492b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.893410 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-home" pod "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" (UID: "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.893516 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-model-cache" pod "686a5cc9-26a2-44e2-8a1a-ba5001f492b7" (UID: "686a5cc9-26a2-44e2-8a1a-ba5001f492b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.893713 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-tmp-dir" pod "e441066e-613b-4f47-be65-0e60deb73fc2" (UID: "e441066e-613b-4f47-be65-0e60deb73fc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.893878 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tmp-dir" pod "686a5cc9-26a2-44e2-8a1a-ba5001f492b7" (UID: "686a5cc9-26a2-44e2-8a1a-ba5001f492b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.894822 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-dshm" pod "686a5cc9-26a2-44e2-8a1a-ba5001f492b7" (UID: "686a5cc9-26a2-44e2-8a1a-ba5001f492b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.895072 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tmp-dir" pod "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" (UID: "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.895235 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-model-cache" pod "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" (UID: "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.896062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.895272 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-home" pod "e441066e-613b-4f47-be65-0e60deb73fc2" (UID: "e441066e-613b-4f47-be65-0e60deb73fc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.898162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.896707 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tls-certs" pod "686a5cc9-26a2-44e2-8a1a-ba5001f492b7" (UID: "686a5cc9-26a2-44e2-8a1a-ba5001f492b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:09.898162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.897371 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-dshm" pod "e441066e-613b-4f47-be65-0e60deb73fc2" (UID: "e441066e-613b-4f47-be65-0e60deb73fc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.898162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.897444 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e441066e-613b-4f47-be65-0e60deb73fc2-kube-api-access-ks65v" pod "e441066e-613b-4f47-be65-0e60deb73fc2" (UID: "e441066e-613b-4f47-be65-0e60deb73fc2"). InnerVolumeSpecName "kube-api-access-ks65v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:09.898337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.898207 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-kube-api-access-xjmbr" pod "686a5cc9-26a2-44e2-8a1a-ba5001f492b7" (UID: "686a5cc9-26a2-44e2-8a1a-ba5001f492b7"). InnerVolumeSpecName "kube-api-access-xjmbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:09.902074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.898424 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-kube-api-access-89bsx" pod "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" (UID: "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6"). InnerVolumeSpecName "kube-api-access-89bsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:09.902074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.898670 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e441066e-613b-4f47-be65-0e60deb73fc2-tls-certs" pod "e441066e-613b-4f47-be65-0e60deb73fc2" (UID: "e441066e-613b-4f47-be65-0e60deb73fc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:09.902074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.899132 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tls-certs" pod "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" (UID: "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:09.902074 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.899195 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-dshm" pod "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" (UID: "6faee2de-c73d-486b-9d70-3f5dfcc8d8a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.993351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993316 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993356 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks65v\" (UniqueName: \"kubernetes.io/projected/e441066e-613b-4f47-be65-0e60deb73fc2-kube-api-access-ks65v\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993375 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89bsx\" (UniqueName: \"kubernetes.io/projected/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-kube-api-access-89bsx\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993391 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993405 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993420 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993434 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993447 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993462 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993476 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993490 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e441066e-613b-4f47-be65-0e60deb73fc2-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993504 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993518 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993534 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjmbr\" (UniqueName: \"kubernetes.io/projected/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-kube-api-access-xjmbr\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993549 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993562 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/686a5cc9-26a2-44e2-8a1a-ba5001f492b7-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993574 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e441066e-613b-4f47-be65-0e60deb73fc2-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.993720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:09.993587 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:10.726721 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.726661 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 01 16:29:10.791191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.791156 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" Nov 01 16:29:10.791414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.791194 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs" event={"ID":"686a5cc9-26a2-44e2-8a1a-ba5001f492b7","Type":"ContainerDied","Data":"213293bacfb827ccd6b5934a25ba24de1df604ff6d2316ed122ab8afe34c9aaf"} Nov 01 16:29:10.791699 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.791252 4911 scope.go:122] "RemoveContainer" containerID="bfae6e904d4f5b826f34f3258993dc346f115b98628346a5aaa3acea6582b6fb" Nov 01 16:29:10.793279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.792916 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" Nov 01 16:29:10.793279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.792947 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6" event={"ID":"6faee2de-c73d-486b-9d70-3f5dfcc8d8a6","Type":"ContainerDied","Data":"505b594a900ff9fb62390e9aee6dabd16a2e8c6e076825a9812b53d61cf7d35a"} Nov 01 16:29:10.794847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.794819 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" Nov 01 16:29:10.794953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.794869 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl" event={"ID":"e441066e-613b-4f47-be65-0e60deb73fc2","Type":"ContainerDied","Data":"3f6b6857f7c43764af81f80fc401fc79cc6e9774588488872cc113ad019989da"} Nov 01 16:29:10.806700 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.806673 4911 scope.go:122] "RemoveContainer" containerID="f761c8391753ff28c3edd55997bee44482d57c2d3862a3c574768e316b09e87f" Nov 01 16:29:10.821928 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.821890 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6"] Nov 01 16:29:10.822016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.821967 4911 scope.go:122] "RemoveContainer" containerID="380bcd066cb147c98ca38e4a86fcaa1c93cf6579d36ea78d9f87c34e679246f3" Nov 01 16:29:10.826151 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.826120 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-2-w2-6b0e0f-kserve-6b88cb9c6f-8v4s6"] Nov 01 16:29:10.837888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.837859 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl"] Nov 01 16:29:10.908096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.844003 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/premium-w2-6b0e0f-kserve-779cc8d955-mlcjl"] Nov 01 16:29:10.908096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.904299 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs"] Nov 01 16:29:10.908096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:10.904398 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/distinct-w2-6b0e0f-kserve-6bfbb4fc74-czxjs"] Nov 01 16:29:10.953280 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:29:10.953235 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:29:11.970164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:11.970126 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" path="/var/lib/kubelet/pods/686a5cc9-26a2-44e2-8a1a-ba5001f492b7/volumes" Nov 01 16:29:11.970731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:11.970714 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" path="/var/lib/kubelet/pods/6faee2de-c73d-486b-9d70-3f5dfcc8d8a6/volumes" Nov 01 16:29:11.971390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:11.971369 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" path="/var/lib/kubelet/pods/e441066e-613b-4f47-be65-0e60deb73fc2/volumes" Nov 01 16:29:33.808461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:33.802442 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx"] Nov 01 16:29:33.808461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:33.802693 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" containerID="cri-o://a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc" gracePeriod=30 Nov 01 16:29:39.105659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.105583 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 16:29:39.202393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.202358 4911 generic.go:356] "Generic (PLEG): container finished" podID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerID="a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc" exitCode=0 Nov 01 16:29:39.202568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.202411 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" Nov 01 16:29:39.202568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.202438 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" event={"ID":"a67ab33f-6a56-48a9-a02a-5e156aed78c7","Type":"ContainerDied","Data":"a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc"} Nov 01 16:29:39.202568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.202507 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx" event={"ID":"a67ab33f-6a56-48a9-a02a-5e156aed78c7","Type":"ContainerDied","Data":"83d278bd244c35a5597b6fb2f6c5260e3cbab76360be69a52571b2ec1e3122bd"} Nov 01 16:29:39.202568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.202539 4911 scope.go:122] "RemoveContainer" containerID="a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc" Nov 01 16:29:39.214842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.214820 4911 scope.go:122] "RemoveContainer" containerID="d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0" Nov 01 16:29:39.220421 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220393 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-podinfo\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220485 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istiod-ca-cert\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220549 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220530 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-kube-api-access-mf66b\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-kube-api-access-mf66b\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220570 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-credential-socket\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220655 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-socket\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220703 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-envoy\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220773 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-token\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-token\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220896 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220806 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-certs\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.220896 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220832 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-credential-socket" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.220896 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220890 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-data\") pod \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\" (UID: \"a67ab33f-6a56-48a9-a02a-5e156aed78c7\") " Nov 01 16:29:39.221084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220931 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-socket" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.221084 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.220939 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istiod-ca-cert" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:39.221275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.221244 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-certs" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.221522 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.221496 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-data" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.221768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.221740 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.221859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.221772 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.221859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.221787 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.221859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.221803 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.221859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.221819 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.222907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.222887 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-podinfo" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:29:39.223296 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.223270 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-token" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:39.223425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.223408 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-kube-api-access-mf66b" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "kube-api-access-mf66b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:39.224410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.224390 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-envoy" pod "a67ab33f-6a56-48a9-a02a-5e156aed78c7" (UID: "a67ab33f-6a56-48a9-a02a-5e156aed78c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.226456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.226438 4911 scope.go:122] "RemoveContainer" containerID="a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc" Nov 01 16:29:39.226776 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:29:39.226753 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc\": container with ID starting with a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc not found: ID does not exist" containerID="a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc" Nov 01 16:29:39.226828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.226785 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc"} err="failed to get container status \"a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc\": rpc error: code = NotFound desc = could not find container \"a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc\": container with ID starting with a250dc3c8779273a149ee9253e8aa9cd994e38821ddb59b401e0601c78fdd3bc not found: ID does not exist" Nov 01 16:29:39.226828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.226804 4911 scope.go:122] "RemoveContainer" containerID="d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0" Nov 01 16:29:39.227092 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:29:39.227067 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0\": container with ID starting with d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0 not found: ID does not exist" containerID="d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0" Nov 01 16:29:39.227144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.227101 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0"} err="failed to get container status \"d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0\": rpc error: code = NotFound desc = could not find container \"d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0\": container with ID starting with d04ac876e3e4a69d7d9d2273775c021efbe2bd7f24c40795474beef6cdca6db0 not found: ID does not exist" Nov 01 16:29:39.323316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.323276 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf66b\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-kube-api-access-mf66b\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.323316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.323311 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.323316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.323321 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.323565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.323330 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67ab33f-6a56-48a9-a02a-5e156aed78c7-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.562021 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.561991 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx"] Nov 01 16:29:39.566736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.566707 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-87srx"] Nov 01 16:29:39.984005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:29:39.983918 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" path="/var/lib/kubelet/pods/a67ab33f-6a56-48a9-a02a-5e156aed78c7/volumes" Nov 01 16:29:40.868907 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:29:40.868866 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:30:00.217454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.217411 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5"] Nov 01 16:30:00.218614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218581 4911 cpu_manager.go:414] "containerMap: removing container" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" Nov 01 16:30:00.218614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218612 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218659 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218667 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218684 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218694 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218706 4911 cpu_manager.go:414] "containerMap: removing container" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218714 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218733 4911 cpu_manager.go:414] "containerMap: removing container" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218741 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218779 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218787 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218805 4911 cpu_manager.go:414] "containerMap: removing container" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218813 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" Nov 01 16:30:00.218834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218837 4911 cpu_manager.go:414] "containerMap: removing container" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.218844 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219005 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e441066e-613b-4f47-be65-0e60deb73fc2" containerName="main" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219022 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="6faee2de-c73d-486b-9d70-3f5dfcc8d8a6" containerName="main" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219034 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="5061671b-c676-4375-b9c5-53637897fb5f" containerName="istio-proxy" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219068 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="20770135-5e84-4719-8058-95c73685f02a" containerName="istio-proxy" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219084 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219095 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67ab33f-6a56-48a9-a02a-5e156aed78c7" containerName="istio-proxy" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219105 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="0105a45c-db4b-4929-950e-4714d7ac9369" containerName="istio-proxy" Nov 01 16:30:00.219581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.219117 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="686a5cc9-26a2-44e2-8a1a-ba5001f492b7" containerName="main" Nov 01 16:30:00.225573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.225542 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.238058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.238016 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5"] Nov 01 16:30:00.243343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.243313 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6"] Nov 01 16:30:00.249699 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.249671 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.257960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.257926 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:30:00.271325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.270750 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6"] Nov 01 16:30:00.370308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.370273 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ed2c47-3712-4d25-8327-3592bfdbda73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.370308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.370313 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e661c6a-3713-4fa7-aa98-20c52fc237fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.370590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.370453 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8n6f\" (UniqueName: \"kubernetes.io/projected/8e661c6a-3713-4fa7-aa98-20c52fc237fb-kube-api-access-n8n6f\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.370647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.370627 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzjn\" (UniqueName: \"kubernetes.io/projected/e8ed2c47-3712-4d25-8327-3592bfdbda73-kube-api-access-2zzjn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.472425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.472327 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8n6f\" (UniqueName: \"kubernetes.io/projected/8e661c6a-3713-4fa7-aa98-20c52fc237fb-kube-api-access-n8n6f\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.472611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.472475 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzjn\" (UniqueName: \"kubernetes.io/projected/e8ed2c47-3712-4d25-8327-3592bfdbda73-kube-api-access-2zzjn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.472611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.472605 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ed2c47-3712-4d25-8327-3592bfdbda73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.472802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.472634 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e661c6a-3713-4fa7-aa98-20c52fc237fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.473279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.473255 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ed2c47-3712-4d25-8327-3592bfdbda73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.473413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.473393 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e661c6a-3713-4fa7-aa98-20c52fc237fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.481880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.481853 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzjn\" (UniqueName: \"kubernetes.io/projected/e8ed2c47-3712-4d25-8327-3592bfdbda73-kube-api-access-2zzjn\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.482205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.482185 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8n6f\" (UniqueName: \"kubernetes.io/projected/8e661c6a-3713-4fa7-aa98-20c52fc237fb-kube-api-access-n8n6f\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.549718 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.549680 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:00.573333 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.573300 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:00.835173 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.835137 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5"] Nov 01 16:30:00.835538 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:30:00.835513 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ed2c47_3712_4d25_8327_3592bfdbda73.slice/crio-bec7b54795107fb496a82e013ccb7c89f232c273394f8a70ef7a15151607ab63 WatchSource:0}: Error finding container bec7b54795107fb496a82e013ccb7c89f232c273394f8a70ef7a15151607ab63: Status 404 returned error can't find the container with id bec7b54795107fb496a82e013ccb7c89f232c273394f8a70ef7a15151607ab63 Nov 01 16:30:00.908546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:00.908514 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6"] Nov 01 16:30:00.908964 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:30:00.908940 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e661c6a_3713_4fa7_aa98_20c52fc237fb.slice/crio-31b8275cf136d0c026f923f23f05dc34d1f998df8fbff25852cf5b7a0ced3608 WatchSource:0}: Error finding container 31b8275cf136d0c026f923f23f05dc34d1f998df8fbff25852cf5b7a0ced3608: Status 404 returned error can't find the container with id 31b8275cf136d0c026f923f23f05dc34d1f998df8fbff25852cf5b7a0ced3608 Nov 01 16:30:01.442595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:01.442494 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerStarted","Data":"eaec0eb95ed1687b2e088bf57f3b9ed1659aa6a2a08f3e49b0a5a18b7bee85de"} Nov 01 16:30:01.442595 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:01.442563 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerStarted","Data":"31b8275cf136d0c026f923f23f05dc34d1f998df8fbff25852cf5b7a0ced3608"} Nov 01 16:30:01.444622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:01.444585 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerStarted","Data":"026caee6e27c7dfe3b09be014d742c6670ddc0a70b1a0bdfa33327ec184c267d"} Nov 01 16:30:01.444783 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:01.444638 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerStarted","Data":"bec7b54795107fb496a82e013ccb7c89f232c273394f8a70ef7a15151607ab63"} Nov 01 16:30:01.460800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:01.460748 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" podStartSLOduration=1.460735135 podStartE2EDuration="1.460735135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:01.457993675 +0000 UTC m=+2541.602448315" watchObservedRunningTime="2026-11-01 16:30:01.460735135 +0000 UTC m=+2541.605189807" Nov 01 16:30:01.474458 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:01.474405 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" podStartSLOduration=1.4743906789999999 podStartE2EDuration="1.474390679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:01.472642475 +0000 UTC m=+2541.617097115" watchObservedRunningTime="2026-11-01 16:30:01.474390679 +0000 UTC m=+2541.618845319" Nov 01 16:30:02.453546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:02.453504 4911 generic.go:356] "Generic (PLEG): container finished" podID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerID="eaec0eb95ed1687b2e088bf57f3b9ed1659aa6a2a08f3e49b0a5a18b7bee85de" exitCode=60 Nov 01 16:30:02.453989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:02.453590 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerDied","Data":"eaec0eb95ed1687b2e088bf57f3b9ed1659aa6a2a08f3e49b0a5a18b7bee85de"} Nov 01 16:30:02.453989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:02.453927 4911 scope.go:122] "RemoveContainer" containerID="eaec0eb95ed1687b2e088bf57f3b9ed1659aa6a2a08f3e49b0a5a18b7bee85de" Nov 01 16:30:02.455648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:02.455623 4911 generic.go:356] "Generic (PLEG): container finished" podID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerID="026caee6e27c7dfe3b09be014d742c6670ddc0a70b1a0bdfa33327ec184c267d" exitCode=60 Nov 01 16:30:02.455772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:02.455708 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerDied","Data":"026caee6e27c7dfe3b09be014d742c6670ddc0a70b1a0bdfa33327ec184c267d"} Nov 01 16:30:02.456199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:02.456176 4911 scope.go:122] "RemoveContainer" containerID="026caee6e27c7dfe3b09be014d742c6670ddc0a70b1a0bdfa33327ec184c267d" Nov 01 16:30:03.465355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:03.465311 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerStarted","Data":"5c059856bb9773f460ef54d741e344502feaa6f0285d67706e8f6467cefb8932"} Nov 01 16:30:03.467339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:03.467310 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerStarted","Data":"9eedc7a6c48b952a2e542b6d851d6780b8dff0db2dad4746b9634a5ba0e18501"} Nov 01 16:30:04.477561 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.477517 4911 generic.go:356] "Generic (PLEG): container finished" podID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerID="5c059856bb9773f460ef54d741e344502feaa6f0285d67706e8f6467cefb8932" exitCode=60 Nov 01 16:30:04.478092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.477602 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerDied","Data":"5c059856bb9773f460ef54d741e344502feaa6f0285d67706e8f6467cefb8932"} Nov 01 16:30:04.478092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.477667 4911 scope.go:122] "RemoveContainer" containerID="eaec0eb95ed1687b2e088bf57f3b9ed1659aa6a2a08f3e49b0a5a18b7bee85de" Nov 01 16:30:04.478092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.477966 4911 scope.go:122] "RemoveContainer" containerID="5c059856bb9773f460ef54d741e344502feaa6f0285d67706e8f6467cefb8932" Nov 01 16:30:04.478404 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:04.478378 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6_odh-ai-gateway-infra(8e661c6a-3713-4fa7-aa98-20c52fc237fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" Nov 01 16:30:04.485585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.481569 4911 generic.go:356] "Generic (PLEG): container finished" podID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerID="9eedc7a6c48b952a2e542b6d851d6780b8dff0db2dad4746b9634a5ba0e18501" exitCode=60 Nov 01 16:30:04.485585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.481748 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerDied","Data":"9eedc7a6c48b952a2e542b6d851d6780b8dff0db2dad4746b9634a5ba0e18501"} Nov 01 16:30:04.485585 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.482165 4911 scope.go:122] "RemoveContainer" containerID="9eedc7a6c48b952a2e542b6d851d6780b8dff0db2dad4746b9634a5ba0e18501" Nov 01 16:30:04.485585 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:04.482524 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5_odh-ai-gateway-infra(e8ed2c47-3712-4d25-8327-3592bfdbda73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" Nov 01 16:30:04.497895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.497875 4911 scope.go:122] "RemoveContainer" containerID="026caee6e27c7dfe3b09be014d742c6670ddc0a70b1a0bdfa33327ec184c267d" Nov 01 16:30:04.699064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.685083 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll"] Nov 01 16:30:04.699064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.697303 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.721100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.713178 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:30:04.729295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.729219 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll"] Nov 01 16:30:04.783645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.783537 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdj94\" (UniqueName: \"kubernetes.io/projected/111ff9b5-7933-469c-a4a4-f8645ffc2646-kube-api-access-pdj94\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.784028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.784002 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.784275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.784258 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.886719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.886674 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdj94\" (UniqueName: \"kubernetes.io/projected/111ff9b5-7933-469c-a4a4-f8645ffc2646-kube-api-access-pdj94\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.886923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.886819 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.886923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.886881 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.887718 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.887685 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.890626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.890598 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:04.899640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:04.899608 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdj94\" (UniqueName: \"kubernetes.io/projected/111ff9b5-7933-469c-a4a4-f8645ffc2646-kube-api-access-pdj94\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:05.054722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:05.050557 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:05.341291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:05.341253 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll"] Nov 01 16:30:05.341452 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:30:05.341381 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111ff9b5_7933_469c_a4a4_f8645ffc2646.slice/crio-9298c45a504333ae7f9cae4720efda28581f3d2638faf934ad0d1b53646ad98d WatchSource:0}: Error finding container 9298c45a504333ae7f9cae4720efda28581f3d2638faf934ad0d1b53646ad98d: Status 404 returned error can't find the container with id 9298c45a504333ae7f9cae4720efda28581f3d2638faf934ad0d1b53646ad98d Nov 01 16:30:05.608114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:05.607980 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" event={"ID":"111ff9b5-7933-469c-a4a4-f8645ffc2646","Type":"ContainerStarted","Data":"9298c45a504333ae7f9cae4720efda28581f3d2638faf934ad0d1b53646ad98d"} Nov 01 16:30:05.644065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:05.643906 4911 scope.go:122] "RemoveContainer" containerID="5c059856bb9773f460ef54d741e344502feaa6f0285d67706e8f6467cefb8932" Nov 01 16:30:05.648060 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:05.644301 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6_odh-ai-gateway-infra(8e661c6a-3713-4fa7-aa98-20c52fc237fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" Nov 01 16:30:05.674533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:05.674225 4911 scope.go:122] "RemoveContainer" containerID="9eedc7a6c48b952a2e542b6d851d6780b8dff0db2dad4746b9634a5ba0e18501" Nov 01 16:30:05.674722 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:05.674617 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5_odh-ai-gateway-infra(e8ed2c47-3712-4d25-8327-3592bfdbda73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" Nov 01 16:30:06.244287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:06.243982 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz"] Nov 01 16:30:06.244723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:06.244259 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" containerID="cri-o://c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7" gracePeriod=30 Nov 01 16:30:06.684875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:06.684838 4911 generic.go:356] "Generic (PLEG): container finished" podID="111ff9b5-7933-469c-a4a4-f8645ffc2646" containerID="d954f47e6316cf7881da6e0a5ceddd9f9bcb3982de7ce2cd6908fb62cbddbdcc" exitCode=0 Nov 01 16:30:06.685343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:06.685160 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" event={"ID":"111ff9b5-7933-469c-a4a4-f8645ffc2646","Type":"ContainerDied","Data":"d954f47e6316cf7881da6e0a5ceddd9f9bcb3982de7ce2cd6908fb62cbddbdcc"} Nov 01 16:30:07.965902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:07.965825 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:08.048329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.048294 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111ff9b5-7933-469c-a4a4-f8645ffc2646-kube-api-access-pdj94\" (UniqueName: \"kubernetes.io/projected/111ff9b5-7933-469c-a4a4-f8645ffc2646-kube-api-access-pdj94\") pod \"111ff9b5-7933-469c-a4a4-f8645ffc2646\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " Nov 01 16:30:08.048529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.048336 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-openshift-ca\") pod \"111ff9b5-7933-469c-a4a4-f8645ffc2646\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " Nov 01 16:30:08.048529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.048488 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-xks-ca\") pod \"111ff9b5-7933-469c-a4a4-f8645ffc2646\" (UID: \"111ff9b5-7933-469c-a4a4-f8645ffc2646\") " Nov 01 16:30:08.048838 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.048811 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-openshift-ca" pod "111ff9b5-7933-469c-a4a4-f8645ffc2646" (UID: "111ff9b5-7933-469c-a4a4-f8645ffc2646"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:08.049279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.049255 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:08.050562 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.050540 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111ff9b5-7933-469c-a4a4-f8645ffc2646-kube-api-access-pdj94" pod "111ff9b5-7933-469c-a4a4-f8645ffc2646" (UID: "111ff9b5-7933-469c-a4a4-f8645ffc2646"). InnerVolumeSpecName "kube-api-access-pdj94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:08.050654 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.050621 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-xks-ca" pod "111ff9b5-7933-469c-a4a4-f8645ffc2646" (UID: "111ff9b5-7933-469c-a4a4-f8645ffc2646"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:08.150673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.150636 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111ff9b5-7933-469c-a4a4-f8645ffc2646-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:08.150673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.150670 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdj94\" (UniqueName: \"kubernetes.io/projected/111ff9b5-7933-469c-a4a4-f8645ffc2646-kube-api-access-pdj94\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:08.702461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.702428 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" Nov 01 16:30:08.702642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.702471 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll" event={"ID":"111ff9b5-7933-469c-a4a4-f8645ffc2646","Type":"ContainerDied","Data":"9298c45a504333ae7f9cae4720efda28581f3d2638faf934ad0d1b53646ad98d"} Nov 01 16:30:08.702642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:08.702520 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9298c45a504333ae7f9cae4720efda28581f3d2638faf934ad0d1b53646ad98d" Nov 01 16:30:10.881950 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:10.881913 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:30:11.681516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.681486 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:30:11.729686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.729642 4911 generic.go:356] "Generic (PLEG): container finished" podID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerID="c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7" exitCode=0 Nov 01 16:30:11.729890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.729703 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" Nov 01 16:30:11.729890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.729758 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" event={"ID":"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81","Type":"ContainerDied","Data":"c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7"} Nov 01 16:30:11.729890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.729826 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz" event={"ID":"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81","Type":"ContainerDied","Data":"28c668c23213df1d59b29747e3f1b0fcfb6165c03316cc67b8e29f6fb590f23f"} Nov 01 16:30:11.729890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.729862 4911 scope.go:122] "RemoveContainer" containerID="c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7" Nov 01 16:30:11.743898 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.743867 4911 scope.go:122] "RemoveContainer" containerID="c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7" Nov 01 16:30:11.744243 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:11.744219 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7\": container with ID starting with c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7 not found: ID does not exist" containerID="c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7" Nov 01 16:30:11.744303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.744257 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7"} err="failed to get container status \"c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7\": rpc error: code = NotFound desc = could not find container \"c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7\": container with ID starting with c62c475d2433933a00b08c1540e4c7c10e3cf018523243abbe6ab57b3ed8d9d7 not found: ID does not exist" Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.847585 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-certs\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.847733 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-data\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.847797 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istiod-ca-cert\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.847912 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-envoy\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.847954 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-kube-api-access-w68xc\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-kube-api-access-w68xc\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.848015 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-podinfo\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.848084 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-credential-socket\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.848137 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-socket\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.856122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.848280 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-token\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-token\") pod \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\" (UID: \"4f27b0b4-d63f-45c4-8408-f67a0a6eaf81\") " Nov 01 16:30:11.859445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.857866 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-socket" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:11.859879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.859822 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-certs" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:11.861568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.861503 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-data" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:11.861716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.861693 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istiod-ca-cert" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:11.862055 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.862013 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-credential-socket" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:11.873952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.873803 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-envoy" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:11.875285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.875229 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-podinfo" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:30:11.878230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.878203 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-token" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:11.878336 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.878305 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-kube-api-access-w68xc" pod "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" (UID: "4f27b0b4-d63f-45c4-8408-f67a0a6eaf81"). InnerVolumeSpecName "kube-api-access-w68xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950347 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950377 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950388 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950398 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w68xc\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-kube-api-access-w68xc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950408 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950419 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950430 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.950434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950439 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:11.951030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:11.950448 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:12.051364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:12.051296 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz"] Nov 01 16:30:12.055864 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:12.055832 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5e2d03-openshift-default-577cbdbb4f-vzvrz"] Nov 01 16:30:13.969202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:13.969169 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" path="/var/lib/kubelet/pods/4f27b0b4-d63f-45c4-8408-f67a0a6eaf81/volumes" Nov 01 16:30:14.812853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.812816 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7"] Nov 01 16:30:14.813668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.813644 4911 cpu_manager.go:414] "containerMap: removing container" podUID="111ff9b5-7933-469c-a4a4-f8645ffc2646" containerName="revoke-keys" Nov 01 16:30:14.813732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.813670 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111ff9b5-7933-469c-a4a4-f8645ffc2646" containerName="revoke-keys" Nov 01 16:30:14.813732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.813688 4911 cpu_manager.go:414] "containerMap: removing container" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" Nov 01 16:30:14.813732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.813695 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" Nov 01 16:30:14.813835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.813819 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="111ff9b5-7933-469c-a4a4-f8645ffc2646" containerName="revoke-keys" Nov 01 16:30:14.813835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.813830 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f27b0b4-d63f-45c4-8408-f67a0a6eaf81" containerName="istio-proxy" Nov 01 16:30:14.819956 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.819925 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.823190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.823160 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7"] Nov 01 16:30:14.823823 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.823801 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:30:14.887255 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.887217 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8p8\" (UniqueName: \"kubernetes.io/projected/d5e436d9-9dde-41f3-99d7-4086a17530df-kube-api-access-7w8p8\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.887462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.887436 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.887544 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.887488 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.959296 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.959265 4911 scope.go:122] "RemoveContainer" containerID="9eedc7a6c48b952a2e542b6d851d6780b8dff0db2dad4746b9634a5ba0e18501" Nov 01 16:30:14.989312 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.989272 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8p8\" (UniqueName: \"kubernetes.io/projected/d5e436d9-9dde-41f3-99d7-4086a17530df-kube-api-access-7w8p8\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.989731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.989502 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.989731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.989544 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.990654 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.990623 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.992308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.992280 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:14.997536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:14.997508 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8p8\" (UniqueName: \"kubernetes.io/projected/d5e436d9-9dde-41f3-99d7-4086a17530df-kube-api-access-7w8p8\") pod \"maas-api-revoke-sub-d35e1bb7-wzwt7\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:15.139666 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:15.139574 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:15.430990 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:15.430959 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7"] Nov 01 16:30:15.431997 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:30:15.431969 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e436d9_9dde_41f3_99d7_4086a17530df.slice/crio-0bfda8ba3ffaf3592e73c1fb898a7e8c835ac248b45a64a163ad43e23e08b3d0 WatchSource:0}: Error finding container 0bfda8ba3ffaf3592e73c1fb898a7e8c835ac248b45a64a163ad43e23e08b3d0: Status 404 returned error can't find the container with id 0bfda8ba3ffaf3592e73c1fb898a7e8c835ac248b45a64a163ad43e23e08b3d0 Nov 01 16:30:15.765744 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:15.765711 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerStarted","Data":"a9988763cde6a89ea72621c3af239e2910b1c1ac0046ecafd002d97be6d84266"} Nov 01 16:30:15.767656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:15.767628 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" event={"ID":"d5e436d9-9dde-41f3-99d7-4086a17530df","Type":"ContainerStarted","Data":"0bfda8ba3ffaf3592e73c1fb898a7e8c835ac248b45a64a163ad43e23e08b3d0"} Nov 01 16:30:16.782013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.781967 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn"] Nov 01 16:30:16.783154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.783131 4911 generic.go:356] "Generic (PLEG): container finished" podID="d5e436d9-9dde-41f3-99d7-4086a17530df" containerID="eb861834938f451641faf714d9ae2aff5e8b00cddf395aa22ec34b0753bbcedd" exitCode=0 Nov 01 16:30:16.783231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.783208 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" event={"ID":"d5e436d9-9dde-41f3-99d7-4086a17530df","Type":"ContainerDied","Data":"eb861834938f451641faf714d9ae2aff5e8b00cddf395aa22ec34b0753bbcedd"} Nov 01 16:30:16.787701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.787673 4911 generic.go:356] "Generic (PLEG): container finished" podID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerID="a9988763cde6a89ea72621c3af239e2910b1c1ac0046ecafd002d97be6d84266" exitCode=60 Nov 01 16:30:16.787843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.787712 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerDied","Data":"a9988763cde6a89ea72621c3af239e2910b1c1ac0046ecafd002d97be6d84266"} Nov 01 16:30:16.787843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.787763 4911 scope.go:122] "RemoveContainer" containerID="9eedc7a6c48b952a2e542b6d851d6780b8dff0db2dad4746b9634a5ba0e18501" Nov 01 16:30:16.788529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.788472 4911 scope.go:122] "RemoveContainer" containerID="a9988763cde6a89ea72621c3af239e2910b1c1ac0046ecafd002d97be6d84266" Nov 01 16:30:16.788967 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:16.788938 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5_odh-ai-gateway-infra(e8ed2c47-3712-4d25-8327-3592bfdbda73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" Nov 01 16:30:16.817144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.816296 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5"] Nov 01 16:30:16.959115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:16.959083 4911 scope.go:122] "RemoveContainer" containerID="5c059856bb9773f460ef54d741e344502feaa6f0285d67706e8f6467cefb8932" Nov 01 16:30:17.797761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:17.797710 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerStarted","Data":"020c82be012429ae97b24c60d7858485f17044a81b8516d67a9d31e34b4d7e23"} Nov 01 16:30:18.022928 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.022565 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:18.143129 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.143098 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ed2c47-3712-4d25-8327-3592bfdbda73-kube-api-access-2zzjn\" (UniqueName: \"kubernetes.io/projected/e8ed2c47-3712-4d25-8327-3592bfdbda73-kube-api-access-2zzjn\") pod \"e8ed2c47-3712-4d25-8327-3592bfdbda73\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " Nov 01 16:30:18.143303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.143140 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ed2c47-3712-4d25-8327-3592bfdbda73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ed2c47-3712-4d25-8327-3592bfdbda73-maas-api-service-ca\") pod \"e8ed2c47-3712-4d25-8327-3592bfdbda73\" (UID: \"e8ed2c47-3712-4d25-8327-3592bfdbda73\") " Nov 01 16:30:18.143602 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.143575 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ed2c47-3712-4d25-8327-3592bfdbda73-maas-api-service-ca" pod "e8ed2c47-3712-4d25-8327-3592bfdbda73" (UID: "e8ed2c47-3712-4d25-8327-3592bfdbda73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:18.143936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.143919 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ed2c47-3712-4d25-8327-3592bfdbda73-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.145211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.145189 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ed2c47-3712-4d25-8327-3592bfdbda73-kube-api-access-2zzjn" pod "e8ed2c47-3712-4d25-8327-3592bfdbda73" (UID: "e8ed2c47-3712-4d25-8327-3592bfdbda73"). InnerVolumeSpecName "kube-api-access-2zzjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:18.155790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.155763 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:18.244962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.244910 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-xks-ca\") pod \"d5e436d9-9dde-41f3-99d7-4086a17530df\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " Nov 01 16:30:18.245182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.245086 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e436d9-9dde-41f3-99d7-4086a17530df-kube-api-access-7w8p8\" (UniqueName: \"kubernetes.io/projected/d5e436d9-9dde-41f3-99d7-4086a17530df-kube-api-access-7w8p8\") pod \"d5e436d9-9dde-41f3-99d7-4086a17530df\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " Nov 01 16:30:18.245182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.245146 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-openshift-ca\") pod \"d5e436d9-9dde-41f3-99d7-4086a17530df\" (UID: \"d5e436d9-9dde-41f3-99d7-4086a17530df\") " Nov 01 16:30:18.245467 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.245443 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-openshift-ca" pod "d5e436d9-9dde-41f3-99d7-4086a17530df" (UID: "d5e436d9-9dde-41f3-99d7-4086a17530df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:18.245771 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.245754 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.245820 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.245774 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zzjn\" (UniqueName: \"kubernetes.io/projected/e8ed2c47-3712-4d25-8327-3592bfdbda73-kube-api-access-2zzjn\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.247113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.247079 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e436d9-9dde-41f3-99d7-4086a17530df-kube-api-access-7w8p8" pod "d5e436d9-9dde-41f3-99d7-4086a17530df" (UID: "d5e436d9-9dde-41f3-99d7-4086a17530df"). InnerVolumeSpecName "kube-api-access-7w8p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:18.247236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.247169 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-xks-ca" pod "d5e436d9-9dde-41f3-99d7-4086a17530df" (UID: "d5e436d9-9dde-41f3-99d7-4086a17530df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:18.355809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.353510 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e436d9-9dde-41f3-99d7-4086a17530df-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.355809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.353557 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w8p8\" (UniqueName: \"kubernetes.io/projected/d5e436d9-9dde-41f3-99d7-4086a17530df-kube-api-access-7w8p8\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.815051 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.815006 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" Nov 01 16:30:18.815051 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.815015 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5" event={"ID":"e8ed2c47-3712-4d25-8327-3592bfdbda73","Type":"ContainerDied","Data":"bec7b54795107fb496a82e013ccb7c89f232c273394f8a70ef7a15151607ab63"} Nov 01 16:30:18.815629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.815115 4911 scope.go:122] "RemoveContainer" containerID="a9988763cde6a89ea72621c3af239e2910b1c1ac0046ecafd002d97be6d84266" Nov 01 16:30:18.816809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.816778 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" Nov 01 16:30:18.816936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.816804 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7" event={"ID":"d5e436d9-9dde-41f3-99d7-4086a17530df","Type":"ContainerDied","Data":"0bfda8ba3ffaf3592e73c1fb898a7e8c835ac248b45a64a163ad43e23e08b3d0"} Nov 01 16:30:18.816936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.816858 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bfda8ba3ffaf3592e73c1fb898a7e8c835ac248b45a64a163ad43e23e08b3d0" Nov 01 16:30:18.822178 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.822148 4911 generic.go:356] "Generic (PLEG): container finished" podID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerID="020c82be012429ae97b24c60d7858485f17044a81b8516d67a9d31e34b4d7e23" exitCode=60 Nov 01 16:30:18.822314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.822215 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerDied","Data":"020c82be012429ae97b24c60d7858485f17044a81b8516d67a9d31e34b4d7e23"} Nov 01 16:30:18.822708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.822689 4911 scope.go:122] "RemoveContainer" containerID="020c82be012429ae97b24c60d7858485f17044a81b8516d67a9d31e34b4d7e23" Nov 01 16:30:18.823101 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:18.823016 4911 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6_odh-ai-gateway-infra(8e661c6a-3713-4fa7-aa98-20c52fc237fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" Nov 01 16:30:18.830000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.829906 4911 scope.go:122] "RemoveContainer" containerID="5c059856bb9773f460ef54d741e344502feaa6f0285d67706e8f6467cefb8932" Nov 01 16:30:18.832853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.832833 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6"] Nov 01 16:30:18.866851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.866819 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5"] Nov 01 16:30:18.871732 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:18.871704 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-383fde-29892510-xs7q5"] Nov 01 16:30:19.974815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:19.974770 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" path="/var/lib/kubelet/pods/e8ed2c47-3712-4d25-8327-3592bfdbda73/volumes" Nov 01 16:30:20.019502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.019476 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:20.086876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.078838 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e661c6a-3713-4fa7-aa98-20c52fc237fb-kube-api-access-n8n6f\" (UniqueName: \"kubernetes.io/projected/8e661c6a-3713-4fa7-aa98-20c52fc237fb-kube-api-access-n8n6f\") pod \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " Nov 01 16:30:20.086876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.084260 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e661c6a-3713-4fa7-aa98-20c52fc237fb-kube-api-access-n8n6f" pod "8e661c6a-3713-4fa7-aa98-20c52fc237fb" (UID: "8e661c6a-3713-4fa7-aa98-20c52fc237fb"). InnerVolumeSpecName "kube-api-access-n8n6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:20.086876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.085208 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e661c6a-3713-4fa7-aa98-20c52fc237fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e661c6a-3713-4fa7-aa98-20c52fc237fb-maas-api-service-ca\") pod \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\" (UID: \"8e661c6a-3713-4fa7-aa98-20c52fc237fb\") " Nov 01 16:30:20.086876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.085658 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e661c6a-3713-4fa7-aa98-20c52fc237fb-maas-api-service-ca" pod "8e661c6a-3713-4fa7-aa98-20c52fc237fb" (UID: "8e661c6a-3713-4fa7-aa98-20c52fc237fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:20.086876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.086185 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8n6f\" (UniqueName: \"kubernetes.io/projected/8e661c6a-3713-4fa7-aa98-20c52fc237fb-kube-api-access-n8n6f\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.086876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.086207 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e661c6a-3713-4fa7-aa98-20c52fc237fb-maas-api-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.875565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.875474 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" Nov 01 16:30:20.875565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.875507 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6" event={"ID":"8e661c6a-3713-4fa7-aa98-20c52fc237fb","Type":"ContainerDied","Data":"31b8275cf136d0c026f923f23f05dc34d1f998df8fbff25852cf5b7a0ced3608"} Nov 01 16:30:20.875747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.875590 4911 scope.go:122] "RemoveContainer" containerID="020c82be012429ae97b24c60d7858485f17044a81b8516d67a9d31e34b4d7e23" Nov 01 16:30:20.903514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.903480 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6"] Nov 01 16:30:20.907688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:20.907667 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892510-9rqq6"] Nov 01 16:30:21.782852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:21.782789 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" podUID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" containerName="payload-processing" containerID="cri-o://8f2492310eb2284af0f0727f6da76104c37ce9aca975a7396427ef0628a505c4" gracePeriod=25 Nov 01 16:30:21.967869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:21.967725 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" path="/var/lib/kubelet/pods/8e661c6a-3713-4fa7-aa98-20c52fc237fb/volumes" Nov 01 16:30:28.679667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:28.679620 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" podUID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:30:29.199564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.199308 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb"] Nov 01 16:30:29.199756 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.199598 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" podUID="c2464d8c-3a41-4040-9087-159740306188" containerName="maas-api" containerID="cri-o://f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3" gracePeriod=30 Nov 01 16:30:29.774086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.774024 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:30:29.804828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.804778 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2464d8c-3a41-4040-9087-159740306188-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2464d8c-3a41-4040-9087-159740306188-openshift-service-ca\") pod \"c2464d8c-3a41-4040-9087-159740306188\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " Nov 01 16:30:29.805029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.804837 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2464d8c-3a41-4040-9087-159740306188-kube-api-access-jj9fz\" (UniqueName: \"kubernetes.io/projected/c2464d8c-3a41-4040-9087-159740306188-kube-api-access-jj9fz\") pod \"c2464d8c-3a41-4040-9087-159740306188\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " Nov 01 16:30:29.805029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.804882 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-maas-api-tls\") pod \"c2464d8c-3a41-4040-9087-159740306188\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " Nov 01 16:30:29.805424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.805106 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-metrics-certs\") pod \"c2464d8c-3a41-4040-9087-159740306188\" (UID: \"c2464d8c-3a41-4040-9087-159740306188\") " Nov 01 16:30:29.805424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.805222 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2464d8c-3a41-4040-9087-159740306188-openshift-service-ca" pod "c2464d8c-3a41-4040-9087-159740306188" (UID: "c2464d8c-3a41-4040-9087-159740306188"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:29.805834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.805810 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2464d8c-3a41-4040-9087-159740306188-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:29.808164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.808135 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2464d8c-3a41-4040-9087-159740306188-kube-api-access-jj9fz" pod "c2464d8c-3a41-4040-9087-159740306188" (UID: "c2464d8c-3a41-4040-9087-159740306188"). InnerVolumeSpecName "kube-api-access-jj9fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:29.808890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.808863 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-metrics-certs" pod "c2464d8c-3a41-4040-9087-159740306188" (UID: "c2464d8c-3a41-4040-9087-159740306188"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:29.810270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.810245 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-maas-api-tls" pod "c2464d8c-3a41-4040-9087-159740306188" (UID: "c2464d8c-3a41-4040-9087-159740306188"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:29.923068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.919089 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj9fz\" (UniqueName: \"kubernetes.io/projected/c2464d8c-3a41-4040-9087-159740306188-kube-api-access-jj9fz\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:29.923068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.919134 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:29.923068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.919150 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2464d8c-3a41-4040-9087-159740306188-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:29.998797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.998762 4911 generic.go:356] "Generic (PLEG): container finished" podID="c2464d8c-3a41-4040-9087-159740306188" containerID="f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3" exitCode=0 Nov 01 16:30:29.998967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.998808 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" Nov 01 16:30:29.998967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.998847 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" event={"ID":"c2464d8c-3a41-4040-9087-159740306188","Type":"ContainerDied","Data":"f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3"} Nov 01 16:30:29.998967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.998914 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb" event={"ID":"c2464d8c-3a41-4040-9087-159740306188","Type":"ContainerDied","Data":"8878811743e966d3072f653a23b0a4ecc6e33b41006e08d63428c959225239f0"} Nov 01 16:30:29.998967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:29.998948 4911 scope.go:122] "RemoveContainer" containerID="f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3" Nov 01 16:30:30.013339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:30.013310 4911 scope.go:122] "RemoveContainer" containerID="f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3" Nov 01 16:30:30.013631 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:30.013611 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3\": container with ID starting with f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3 not found: ID does not exist" containerID="f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3" Nov 01 16:30:30.013725 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:30.013640 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3"} err="failed to get container status \"f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3\": rpc error: code = NotFound desc = could not find container \"f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3\": container with ID starting with f0c417971d252d112a9cb149f97b4aa03fcefd2bc2f415dcbc3b2d79371cdfc3 not found: ID does not exist" Nov 01 16:30:30.025794 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:30.025768 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb"] Nov 01 16:30:30.029576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:30.029552 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d121-5f6547b79b-drsbb"] Nov 01 16:30:31.969108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:31.969072 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2464d8c-3a41-4040-9087-159740306188" path="/var/lib/kubelet/pods/c2464d8c-3a41-4040-9087-159740306188/volumes" Nov 01 16:30:35.182407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:35.182337 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll"] Nov 01 16:30:35.188619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:35.188584 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1e630-6ttll"] Nov 01 16:30:36.015033 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:36.014983 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111ff9b5-7933-469c-a4a4-f8645ffc2646" path="/var/lib/kubelet/pods/111ff9b5-7933-469c-a4a4-f8645ffc2646/volumes" Nov 01 16:30:37.549910 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:37.549872 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv"] Nov 01 16:30:37.551088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:37.551005 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" containerID="cri-o://b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62" gracePeriod=30 Nov 01 16:30:38.678473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:38.678426 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" podUID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:30:40.867654 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:40.867614 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:30:42.167701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.167382 4911 generic.go:356] "Generic (PLEG): container finished" podID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" containerID="8f2492310eb2284af0f0727f6da76104c37ce9aca975a7396427ef0628a505c4" exitCode=0 Nov 01 16:30:42.167701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.167477 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" event={"ID":"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e","Type":"ContainerDied","Data":"8f2492310eb2284af0f0727f6da76104c37ce9aca975a7396427ef0628a505c4"} Nov 01 16:30:42.235868 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.235843 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:30:42.337672 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.337639 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-plugins-config-volume\") pod \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " Nov 01 16:30:42.337863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.337690 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-kube-api-access-qbz4l\" (UniqueName: \"kubernetes.io/projected/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-kube-api-access-qbz4l\") pod \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\" (UID: \"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e\") " Nov 01 16:30:42.338079 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.338035 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-plugins-config-volume" pod "6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" (UID: "6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:42.338236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.338218 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.340002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.339977 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-kube-api-access-qbz4l" pod "6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" (UID: "6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e"). InnerVolumeSpecName "kube-api-access-qbz4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:42.440291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.440250 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbz4l\" (UniqueName: \"kubernetes.io/projected/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e-kube-api-access-qbz4l\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.869872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.869843 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:30:42.945866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.945767 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-credential-socket\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.945892 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-socket\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.945993 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-kube-api-access-zhqzc\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-kube-api-access-zhqzc\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946282 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946100 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-envoy\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946282 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946178 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-podinfo\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946282 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946241 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-certs\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946282 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946278 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istiod-ca-cert\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946174 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-credential-socket" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:42.946512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946314 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-data\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946365 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-socket" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:42.946512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946407 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-token\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-token\") pod \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\" (UID: \"a74269e5-3eee-4c31-acf1-a610e61b8ed3\") " Nov 01 16:30:42.946706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946681 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-data" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:42.946811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.946707 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istiod-ca-cert" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:42.947089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.947067 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.947264 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.947240 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.947399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.947266 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.947399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.947283 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.947399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.947175 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-certs" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:42.948800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.948775 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-envoy" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:42.948906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.948836 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-podinfo" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:30:42.948971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.948934 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-token" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:42.949031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.948976 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-kube-api-access-zhqzc" pod "a74269e5-3eee-4c31-acf1-a610e61b8ed3" (UID: "a74269e5-3eee-4c31-acf1-a610e61b8ed3"). InnerVolumeSpecName "kube-api-access-zhqzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:42.968598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:42.968558 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs"] Nov 01 16:30:43.048404 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.048363 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.048404 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.048396 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.048404 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.048406 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhqzc\" (UniqueName: \"kubernetes.io/projected/a74269e5-3eee-4c31-acf1-a610e61b8ed3-kube-api-access-zhqzc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.048651 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.048416 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.048651 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.048426 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74269e5-3eee-4c31-acf1-a610e61b8ed3-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:43.177542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.177503 4911 generic.go:356] "Generic (PLEG): container finished" podID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerID="b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62" exitCode=0 Nov 01 16:30:43.178061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.177551 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" Nov 01 16:30:43.178061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.177573 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" event={"ID":"a74269e5-3eee-4c31-acf1-a610e61b8ed3","Type":"ContainerDied","Data":"b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62"} Nov 01 16:30:43.178061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.177634 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv" event={"ID":"a74269e5-3eee-4c31-acf1-a610e61b8ed3","Type":"ContainerDied","Data":"33ce68b5b34b0d7b1f2cd216c5027540152fbc73ab3c7058009b442b59618f9e"} Nov 01 16:30:43.178061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.177659 4911 scope.go:122] "RemoveContainer" containerID="b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62" Nov 01 16:30:43.179505 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.179484 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" Nov 01 16:30:43.179640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.179511 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn" event={"ID":"6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e","Type":"ContainerDied","Data":"c84041df5992f585c21fe11c82783703139a89c6ba4e17774f4edad33b74cf17"} Nov 01 16:30:43.191853 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.191828 4911 scope.go:122] "RemoveContainer" containerID="ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45" Nov 01 16:30:43.205150 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.205128 4911 scope.go:122] "RemoveContainer" containerID="b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62" Nov 01 16:30:43.205443 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:43.205425 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62\": container with ID starting with b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62 not found: ID does not exist" containerID="b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62" Nov 01 16:30:43.205499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.205450 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62"} err="failed to get container status \"b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62\": rpc error: code = NotFound desc = could not find container \"b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62\": container with ID starting with b1c148a97487476777bd4b7e8ee7e3a0b3cc6352e5a205af6afa8d6a0c13cb62 not found: ID does not exist" Nov 01 16:30:43.205499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.205466 4911 scope.go:122] "RemoveContainer" containerID="ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45" Nov 01 16:30:43.205685 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:43.205665 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45\": container with ID starting with ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45 not found: ID does not exist" containerID="ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45" Nov 01 16:30:43.205793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.205685 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45"} err="failed to get container status \"ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45\": rpc error: code = NotFound desc = could not find container \"ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45\": container with ID starting with ababf73b0aa0820c108e778e61535269a6f67ff23f2b77a98afe59e446138b45 not found: ID does not exist" Nov 01 16:30:43.205793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.205696 4911 scope.go:122] "RemoveContainer" containerID="8f2492310eb2284af0f0727f6da76104c37ce9aca975a7396427ef0628a505c4" Nov 01 16:30:43.213770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.213732 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv"] Nov 01 16:30:43.217880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.217855 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d121-openshift-default-786dfd7b4c-ncnzv"] Nov 01 16:30:43.226480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.226458 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn"] Nov 01 16:30:43.229865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.229843 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d121-66bb6ff85-8mssn"] Nov 01 16:30:43.969092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.969055 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" path="/var/lib/kubelet/pods/6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e/volumes" Nov 01 16:30:43.969710 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:43.969693 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" path="/var/lib/kubelet/pods/a74269e5-3eee-4c31-acf1-a610e61b8ed3/volumes" Nov 01 16:30:44.314164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314126 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2"] Nov 01 16:30:44.314754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314740 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" Nov 01 16:30:44.314796 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314754 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" Nov 01 16:30:44.314796 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314774 4911 cpu_manager.go:414] "containerMap: removing container" podUID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" containerName="payload-processing" Nov 01 16:30:44.314796 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314779 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" containerName="payload-processing" Nov 01 16:30:44.314796 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314794 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314799 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314808 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314812 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314819 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d5e436d9-9dde-41f3-99d7-4086a17530df" containerName="revoke-keys" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314824 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e436d9-9dde-41f3-99d7-4086a17530df" containerName="revoke-keys" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314832 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314836 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314845 4911 cpu_manager.go:414] "containerMap: removing container" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314849 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314855 4911 cpu_manager.go:414] "containerMap: removing container" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314860 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314866 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314871 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314881 4911 cpu_manager.go:414] "containerMap: removing container" podUID="c2464d8c-3a41-4040-9087-159740306188" containerName="maas-api" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314886 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2464d8c-3a41-4040-9087-159740306188" containerName="maas-api" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314902 4911 cpu_manager.go:414] "containerMap: removing container" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.314924 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.314907 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315007 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315016 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315025 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e436d9-9dde-41f3-99d7-4086a17530df" containerName="revoke-keys" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315031 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315053 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315059 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315065 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ed2c47-3712-4d25-8327-3592bfdbda73" containerName="cleanup" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315071 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2464d8c-3a41-4040-9087-159740306188" containerName="maas-api" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315079 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e661c6a-3713-4fa7-aa98-20c52fc237fb" containerName="cleanup" Nov 01 16:30:44.315437 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.315090 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0631e6-9b1d-4b0d-b99c-96036c6e0f0e" containerName="payload-processing" Nov 01 16:30:44.320287 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.320266 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.327015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.326979 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-028c46-openshift-default-dockercfg-8z222\"" Nov 01 16:30:44.333302 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.333277 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2"] Nov 01 16:30:44.368008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.367968 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368027 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368163 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a01b32b-d146-4929-9465-27b0aa46a09c-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368268 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368298 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368396 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368324 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db99h\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-kube-api-access-db99h\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368396 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368350 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368439 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.368512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.368497 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.469673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.469630 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.469882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.469709 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.469882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.469798 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.469882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.469831 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.469926 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a01b32b-d146-4929-9465-27b0aa46a09c-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.469999 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470021 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470052 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db99h\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-kube-api-access-db99h\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470078 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470378 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470127 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470398 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470656 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470633 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470730 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470658 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.470826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.470800 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a01b32b-d146-4929-9465-27b0aa46a09c-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.472608 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.472586 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.473283 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.473249 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.478214 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.478191 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.478951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.478917 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db99h\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-kube-api-access-db99h\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.641258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.639880 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:44.941249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.941216 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2"] Nov 01 16:30:44.941955 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:30:44.941924 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a01b32b_d146_4929_9465_27b0aa46a09c.slice/crio-481b3da01a3f8ff157c0458aa790b8134dde65d6693f57b77fbe5e9a03f82a0d WatchSource:0}: Error finding container 481b3da01a3f8ff157c0458aa790b8134dde65d6693f57b77fbe5e9a03f82a0d: Status 404 returned error can't find the container with id 481b3da01a3f8ff157c0458aa790b8134dde65d6693f57b77fbe5e9a03f82a0d Nov 01 16:30:44.945016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.944954 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:44.945134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.945087 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:44.945134 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:44.945122 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:45.198683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:45.198581 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" event={"ID":"9a01b32b-d146-4929-9465-27b0aa46a09c","Type":"ContainerStarted","Data":"b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06"} Nov 01 16:30:45.198683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:45.198646 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" event={"ID":"9a01b32b-d146-4929-9465-27b0aa46a09c","Type":"ContainerStarted","Data":"481b3da01a3f8ff157c0458aa790b8134dde65d6693f57b77fbe5e9a03f82a0d"} Nov 01 16:30:45.218412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:45.218348 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" podStartSLOduration=1.218330586 podStartE2EDuration="1.218330586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:45.217501092 +0000 UTC m=+2585.361955746" watchObservedRunningTime="2026-11-01 16:30:45.218330586 +0000 UTC m=+2585.362785229" Nov 01 16:30:45.640800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:45.640759 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:46.642954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:46.642903 4911 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 16:30:46.643406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:46.642964 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 16:30:47.641346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:47.641303 4911 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 16:30:47.641714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:47.641380 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 16:30:48.650066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:48.646162 4911 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 16:30:48.650066 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:48.646231 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 16:30:49.641244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:49.641204 4911 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 16:30:49.641428 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:49.641263 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 16:30:50.641247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:50.641205 4911 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 01 16:30:50.641696 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:50.641269 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 01 16:30:51.646218 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:51.646182 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:51.646698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:51.646676 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:51.647487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:51.647469 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:30:57.995110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:57.995069 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" containerID="cri-o://49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554" gracePeriod=45 Nov 01 16:30:58.296923 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.296898 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:30:58.305360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.305333 4911 generic.go:356] "Generic (PLEG): container finished" podID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerID="49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554" exitCode=0 Nov 01 16:30:58.305517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.305383 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" Nov 01 16:30:58.305517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.305423 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" event={"ID":"b523b317-2c74-4159-a618-ce4a6d9635c5","Type":"ContainerDied","Data":"49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554"} Nov 01 16:30:58.305517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.305464 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs" event={"ID":"b523b317-2c74-4159-a618-ce4a6d9635c5","Type":"ContainerDied","Data":"1e056b844078afc2719cd4e9315f6a1e584dff32b1475ad2aece240ebb95022e"} Nov 01 16:30:58.305517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.305487 4911 scope.go:122] "RemoveContainer" containerID="49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554" Nov 01 16:30:58.316420 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.316399 4911 scope.go:122] "RemoveContainer" containerID="49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554" Nov 01 16:30:58.316714 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:30:58.316690 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554\": container with ID starting with 49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554 not found: ID does not exist" containerID="49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554" Nov 01 16:30:58.316927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.316728 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554"} err="failed to get container status \"49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554\": rpc error: code = NotFound desc = could not find container \"49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554\": container with ID starting with 49be079dd03379dfda7fa07c205954d350ac7b7722001b6dad411ce9163dd554 not found: ID does not exist" Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.394267 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-tmp-dir\") pod \"b523b317-2c74-4159-a618-ce4a6d9635c5\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.394434 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-dshm\") pod \"b523b317-2c74-4159-a618-ce4a6d9635c5\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.394476 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b523b317-2c74-4159-a618-ce4a6d9635c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/b523b317-2c74-4159-a618-ce4a6d9635c5-tls-certs\") pod \"b523b317-2c74-4159-a618-ce4a6d9635c5\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.394541 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-home\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-home\") pod \"b523b317-2c74-4159-a618-ce4a6d9635c5\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.394640 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b523b317-2c74-4159-a618-ce4a6d9635c5-kube-api-access-5tq75\" (UniqueName: \"kubernetes.io/projected/b523b317-2c74-4159-a618-ce4a6d9635c5-kube-api-access-5tq75\") pod \"b523b317-2c74-4159-a618-ce4a6d9635c5\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.394679 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-model-cache\") pod \"b523b317-2c74-4159-a618-ce4a6d9635c5\" (UID: \"b523b317-2c74-4159-a618-ce4a6d9635c5\") " Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.395395 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-model-cache" pod "b523b317-2c74-4159-a618-ce4a6d9635c5" (UID: "b523b317-2c74-4159-a618-ce4a6d9635c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.395386 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-home" pod "b523b317-2c74-4159-a618-ce4a6d9635c5" (UID: "b523b317-2c74-4159-a618-ce4a6d9635c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.396083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.395625 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-tmp-dir" pod "b523b317-2c74-4159-a618-ce4a6d9635c5" (UID: "b523b317-2c74-4159-a618-ce4a6d9635c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.399106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.398492 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b523b317-2c74-4159-a618-ce4a6d9635c5-kube-api-access-5tq75" pod "b523b317-2c74-4159-a618-ce4a6d9635c5" (UID: "b523b317-2c74-4159-a618-ce4a6d9635c5"). InnerVolumeSpecName "kube-api-access-5tq75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:58.399106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.398719 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b523b317-2c74-4159-a618-ce4a6d9635c5-tls-certs" pod "b523b317-2c74-4159-a618-ce4a6d9635c5" (UID: "b523b317-2c74-4159-a618-ce4a6d9635c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:58.400163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.400142 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-dshm" pod "b523b317-2c74-4159-a618-ce4a6d9635c5" (UID: "b523b317-2c74-4159-a618-ce4a6d9635c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.497136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.497075 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b523b317-2c74-4159-a618-ce4a6d9635c5-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.497136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.497107 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.497136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.497118 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tq75\" (UniqueName: \"kubernetes.io/projected/b523b317-2c74-4159-a618-ce4a6d9635c5-kube-api-access-5tq75\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.497136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.497130 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.497518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.497157 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.497518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.497166 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b523b317-2c74-4159-a618-ce4a6d9635c5-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.657915 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.657880 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs"] Nov 01 16:30:58.664087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:58.664034 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/distinct-2-w0-4a083d-kserve-55844486cd-679zs"] Nov 01 16:30:59.971388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:30:59.971347 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" path="/var/lib/kubelet/pods/b523b317-2c74-4159-a618-ce4a6d9635c5/volumes" Nov 01 16:31:10.870422 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:31:10.870383 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:31:11.685063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:11.683638 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl"] Nov 01 16:31:11.685063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:11.683887 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" podUID="91743d76-57e1-41b8-af00-9bac5b4ce470" containerName="maas-api" containerID="cri-o://51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9" gracePeriod=30 Nov 01 16:31:12.233529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.233501 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 16:31:12.332203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.332167 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-metrics-certs\") pod \"91743d76-57e1-41b8-af00-9bac5b4ce470\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " Nov 01 16:31:12.332203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.332217 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91743d76-57e1-41b8-af00-9bac5b4ce470-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91743d76-57e1-41b8-af00-9bac5b4ce470-openshift-service-ca\") pod \"91743d76-57e1-41b8-af00-9bac5b4ce470\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " Nov 01 16:31:12.332454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.332279 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-maas-api-tls\") pod \"91743d76-57e1-41b8-af00-9bac5b4ce470\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " Nov 01 16:31:12.332454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.332360 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91743d76-57e1-41b8-af00-9bac5b4ce470-kube-api-access-swtf8\" (UniqueName: \"kubernetes.io/projected/91743d76-57e1-41b8-af00-9bac5b4ce470-kube-api-access-swtf8\") pod \"91743d76-57e1-41b8-af00-9bac5b4ce470\" (UID: \"91743d76-57e1-41b8-af00-9bac5b4ce470\") " Nov 01 16:31:12.332689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.332657 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91743d76-57e1-41b8-af00-9bac5b4ce470-openshift-service-ca" pod "91743d76-57e1-41b8-af00-9bac5b4ce470" (UID: "91743d76-57e1-41b8-af00-9bac5b4ce470"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:12.332883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.332862 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91743d76-57e1-41b8-af00-9bac5b4ce470-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:12.334581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.334556 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91743d76-57e1-41b8-af00-9bac5b4ce470-kube-api-access-swtf8" pod "91743d76-57e1-41b8-af00-9bac5b4ce470" (UID: "91743d76-57e1-41b8-af00-9bac5b4ce470"). InnerVolumeSpecName "kube-api-access-swtf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:12.334685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.334584 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-metrics-certs" pod "91743d76-57e1-41b8-af00-9bac5b4ce470" (UID: "91743d76-57e1-41b8-af00-9bac5b4ce470"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:12.334758 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.334742 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-maas-api-tls" pod "91743d76-57e1-41b8-af00-9bac5b4ce470" (UID: "91743d76-57e1-41b8-af00-9bac5b4ce470"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:12.436621 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.436375 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:12.436621 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.436418 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91743d76-57e1-41b8-af00-9bac5b4ce470-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:12.436621 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.436432 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtf8\" (UniqueName: \"kubernetes.io/projected/91743d76-57e1-41b8-af00-9bac5b4ce470-kube-api-access-swtf8\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:12.474577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.474286 4911 generic.go:356] "Generic (PLEG): container finished" podID="91743d76-57e1-41b8-af00-9bac5b4ce470" containerID="51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9" exitCode=0 Nov 01 16:31:12.474577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.474370 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" event={"ID":"91743d76-57e1-41b8-af00-9bac5b4ce470","Type":"ContainerDied","Data":"51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9"} Nov 01 16:31:12.474577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.474424 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" event={"ID":"91743d76-57e1-41b8-af00-9bac5b4ce470","Type":"ContainerDied","Data":"f586e83315b90922c923bbbf0fce2ca4f8ee4c2e9ad5afce5ce7ece77b2d6e38"} Nov 01 16:31:12.474577 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.474452 4911 scope.go:122] "RemoveContainer" containerID="51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9" Nov 01 16:31:12.474925 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.474640 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl" Nov 01 16:31:12.489488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.489466 4911 scope.go:122] "RemoveContainer" containerID="51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9" Nov 01 16:31:12.489798 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:31:12.489776 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9\": container with ID starting with 51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9 not found: ID does not exist" containerID="51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9" Nov 01 16:31:12.489848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.489808 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9"} err="failed to get container status \"51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9\": rpc error: code = NotFound desc = could not find container \"51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9\": container with ID starting with 51298da7e2e7517ee6e8c98d682e48e7115a26a55567704fd60d7626c5321ce9 not found: ID does not exist" Nov 01 16:31:12.505496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.505021 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl"] Nov 01 16:31:12.509344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:12.509308 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4a083d-79c86f777-cc8kl"] Nov 01 16:31:13.976478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:13.976439 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91743d76-57e1-41b8-af00-9bac5b4ce470" path="/var/lib/kubelet/pods/91743d76-57e1-41b8-af00-9bac5b4ce470/volumes" Nov 01 16:31:40.952065 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:31:40.950994 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:31:50.529714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:50.529672 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8"] Nov 01 16:31:50.530938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:50.530899 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" podUID="52753ffd-ccbf-4250-a1dc-989d4dad113a" containerName="payload-pre-processing" containerID="cri-o://7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250" gracePeriod=30 Nov 01 16:31:51.080024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.079991 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:31:51.117274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.117182 4911 generic.go:356] "Generic (PLEG): container finished" podID="52753ffd-ccbf-4250-a1dc-989d4dad113a" containerID="7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250" exitCode=0 Nov 01 16:31:51.117274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.117239 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" Nov 01 16:31:51.117480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.117275 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" event={"ID":"52753ffd-ccbf-4250-a1dc-989d4dad113a","Type":"ContainerDied","Data":"7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250"} Nov 01 16:31:51.117480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.117336 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8" event={"ID":"52753ffd-ccbf-4250-a1dc-989d4dad113a","Type":"ContainerDied","Data":"ea9d8f11a06e49ea9c9ff2b5f7d987003ffb9f78bcec0163ac8f95fc86b18f2e"} Nov 01 16:31:51.117480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.117366 4911 scope.go:122] "RemoveContainer" containerID="7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250" Nov 01 16:31:51.130106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.129515 4911 scope.go:122] "RemoveContainer" containerID="7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250" Nov 01 16:31:51.130106 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:31:51.129923 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250\": container with ID starting with 7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250 not found: ID does not exist" containerID="7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250" Nov 01 16:31:51.130106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.129968 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250"} err="failed to get container status \"7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250\": rpc error: code = NotFound desc = could not find container \"7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250\": container with ID starting with 7c0cb7541e3eaa4899330a2c3223a8ca957cd1f15edc76d47d4f9356c9259250 not found: ID does not exist" Nov 01 16:31:51.142861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.142827 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52753ffd-ccbf-4250-a1dc-989d4dad113a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52753ffd-ccbf-4250-a1dc-989d4dad113a-plugins-config-volume\") pod \"52753ffd-ccbf-4250-a1dc-989d4dad113a\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " Nov 01 16:31:51.143096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.143076 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52753ffd-ccbf-4250-a1dc-989d4dad113a-kube-api-access-kxc4l\" (UniqueName: \"kubernetes.io/projected/52753ffd-ccbf-4250-a1dc-989d4dad113a-kube-api-access-kxc4l\") pod \"52753ffd-ccbf-4250-a1dc-989d4dad113a\" (UID: \"52753ffd-ccbf-4250-a1dc-989d4dad113a\") " Nov 01 16:31:51.143321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.143293 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52753ffd-ccbf-4250-a1dc-989d4dad113a-plugins-config-volume" pod "52753ffd-ccbf-4250-a1dc-989d4dad113a" (UID: "52753ffd-ccbf-4250-a1dc-989d4dad113a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:51.143689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.143654 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52753ffd-ccbf-4250-a1dc-989d4dad113a-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.145351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.145323 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52753ffd-ccbf-4250-a1dc-989d4dad113a-kube-api-access-kxc4l" pod "52753ffd-ccbf-4250-a1dc-989d4dad113a" (UID: "52753ffd-ccbf-4250-a1dc-989d4dad113a"). InnerVolumeSpecName "kube-api-access-kxc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:51.244759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.244715 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxc4l\" (UniqueName: \"kubernetes.io/projected/52753ffd-ccbf-4250-a1dc-989d4dad113a-kube-api-access-kxc4l\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.442214 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.442178 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8"] Nov 01 16:31:51.447747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.447711 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-56955c97dc-rmhx8"] Nov 01 16:31:51.988227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:31:51.988190 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52753ffd-ccbf-4250-a1dc-989d4dad113a" path="/var/lib/kubelet/pods/52753ffd-ccbf-4250-a1dc-989d4dad113a/volumes" Nov 01 16:32:03.821979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:03.821930 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6"] Nov 01 16:32:03.832673 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:03.832633 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh"] Nov 01 16:32:10.867199 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:10.867156 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:32:13.777086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:13.775158 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926"] Nov 01 16:32:18.777403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:18.777307 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" podUID="3e88ab70-e83d-4abd-95a9-96308c0382d1" containerName="payload-processing" containerID="cri-o://d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51" gracePeriod=25 Nov 01 16:32:18.845975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:18.845923 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" containerID="cri-o://33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3" gracePeriod=45 Nov 01 16:32:18.858728 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:18.858685 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" containerID="cri-o://603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905" gracePeriod=45 Nov 01 16:32:18.912429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:18.912378 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused" Nov 01 16:32:18.966911 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:18.966870 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Nov 01 16:32:19.189933 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.189905 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:32:19.204387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.204359 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281361 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-model-cache\") pod \"783e58a7-525d-43b7-8da4-0be7c41efd4f\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281417 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-home\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-home\") pod \"1cc428f9-1f81-49ae-8aee-5f82900fc489\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281450 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-tmp-dir\") pod \"783e58a7-525d-43b7-8da4-0be7c41efd4f\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281481 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc428f9-1f81-49ae-8aee-5f82900fc489-kube-api-access-ldlkp\" (UniqueName: \"kubernetes.io/projected/1cc428f9-1f81-49ae-8aee-5f82900fc489-kube-api-access-ldlkp\") pod \"1cc428f9-1f81-49ae-8aee-5f82900fc489\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281524 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-home\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-home\") pod \"783e58a7-525d-43b7-8da4-0be7c41efd4f\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281576 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-model-cache\") pod \"1cc428f9-1f81-49ae-8aee-5f82900fc489\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281608 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc428f9-1f81-49ae-8aee-5f82900fc489-tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc428f9-1f81-49ae-8aee-5f82900fc489-tls-certs\") pod \"1cc428f9-1f81-49ae-8aee-5f82900fc489\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281702 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-dshm\") pod \"783e58a7-525d-43b7-8da4-0be7c41efd4f\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281750 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-tmp-dir\") pod \"1cc428f9-1f81-49ae-8aee-5f82900fc489\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281798 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-dshm\") pod \"1cc428f9-1f81-49ae-8aee-5f82900fc489\" (UID: \"1cc428f9-1f81-49ae-8aee-5f82900fc489\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281936 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783e58a7-525d-43b7-8da4-0be7c41efd4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/783e58a7-525d-43b7-8da4-0be7c41efd4f-tls-certs\") pod \"783e58a7-525d-43b7-8da4-0be7c41efd4f\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " Nov 01 16:32:19.282059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.281963 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-model-cache" pod "783e58a7-525d-43b7-8da4-0be7c41efd4f" (UID: "783e58a7-525d-43b7-8da4-0be7c41efd4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.282852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.282102 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783e58a7-525d-43b7-8da4-0be7c41efd4f-kube-api-access-42x8c\" (UniqueName: \"kubernetes.io/projected/783e58a7-525d-43b7-8da4-0be7c41efd4f-kube-api-access-42x8c\") pod \"783e58a7-525d-43b7-8da4-0be7c41efd4f\" (UID: \"783e58a7-525d-43b7-8da4-0be7c41efd4f\") " Nov 01 16:32:19.282852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.282268 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-home" pod "783e58a7-525d-43b7-8da4-0be7c41efd4f" (UID: "783e58a7-525d-43b7-8da4-0be7c41efd4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.282852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.282521 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-model-cache" pod "1cc428f9-1f81-49ae-8aee-5f82900fc489" (UID: "1cc428f9-1f81-49ae-8aee-5f82900fc489"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.282852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.282806 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.282852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.282829 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.282852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.282846 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.283340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.283274 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-home" pod "1cc428f9-1f81-49ae-8aee-5f82900fc489" (UID: "1cc428f9-1f81-49ae-8aee-5f82900fc489"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.283432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.283344 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-tmp-dir" pod "783e58a7-525d-43b7-8da4-0be7c41efd4f" (UID: "783e58a7-525d-43b7-8da4-0be7c41efd4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.284290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.283586 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-tmp-dir" pod "1cc428f9-1f81-49ae-8aee-5f82900fc489" (UID: "1cc428f9-1f81-49ae-8aee-5f82900fc489"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.288390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.288362 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-dshm" pod "783e58a7-525d-43b7-8da4-0be7c41efd4f" (UID: "783e58a7-525d-43b7-8da4-0be7c41efd4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.288390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.288377 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-dshm" pod "1cc428f9-1f81-49ae-8aee-5f82900fc489" (UID: "1cc428f9-1f81-49ae-8aee-5f82900fc489"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.288562 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.288415 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783e58a7-525d-43b7-8da4-0be7c41efd4f-kube-api-access-42x8c" pod "783e58a7-525d-43b7-8da4-0be7c41efd4f" (UID: "783e58a7-525d-43b7-8da4-0be7c41efd4f"). InnerVolumeSpecName "kube-api-access-42x8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:19.288562 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.288488 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc428f9-1f81-49ae-8aee-5f82900fc489-tls-certs" pod "1cc428f9-1f81-49ae-8aee-5f82900fc489" (UID: "1cc428f9-1f81-49ae-8aee-5f82900fc489"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:19.288681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.288583 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc428f9-1f81-49ae-8aee-5f82900fc489-kube-api-access-ldlkp" pod "1cc428f9-1f81-49ae-8aee-5f82900fc489" (UID: "1cc428f9-1f81-49ae-8aee-5f82900fc489"). InnerVolumeSpecName "kube-api-access-ldlkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:19.289874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.289852 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783e58a7-525d-43b7-8da4-0be7c41efd4f-tls-certs" pod "783e58a7-525d-43b7-8da4-0be7c41efd4f" (UID: "783e58a7-525d-43b7-8da4-0be7c41efd4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:19.343135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.343095 4911 generic.go:356] "Generic (PLEG): container finished" podID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerID="603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905" exitCode=0 Nov 01 16:32:19.343135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.343139 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" Nov 01 16:32:19.343411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.343188 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" event={"ID":"1cc428f9-1f81-49ae-8aee-5f82900fc489","Type":"ContainerDied","Data":"603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905"} Nov 01 16:32:19.343411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.343256 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh" event={"ID":"1cc428f9-1f81-49ae-8aee-5f82900fc489","Type":"ContainerDied","Data":"70790a0e4fb8125881eedf73acdf7a6c9dd0047614b52f97536be1a146fdeb32"} Nov 01 16:32:19.343411 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.343283 4911 scope.go:122] "RemoveContainer" containerID="603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905" Nov 01 16:32:19.349576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.349522 4911 generic.go:356] "Generic (PLEG): container finished" podID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerID="33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3" exitCode=0 Nov 01 16:32:19.350314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.349748 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" event={"ID":"783e58a7-525d-43b7-8da4-0be7c41efd4f","Type":"ContainerDied","Data":"33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3"} Nov 01 16:32:19.350314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.349813 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" event={"ID":"783e58a7-525d-43b7-8da4-0be7c41efd4f","Type":"ContainerDied","Data":"ed748f83071d766adfddf2eec761f149434f030c4e5e8cfced407833e667ed9e"} Nov 01 16:32:19.350314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.350076 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6" Nov 01 16:32:19.359412 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.359389 4911 scope.go:122] "RemoveContainer" containerID="603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905" Nov 01 16:32:19.362112 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:19.360299 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905\": container with ID starting with 603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905 not found: ID does not exist" containerID="603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905" Nov 01 16:32:19.362112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.360392 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905"} err="failed to get container status \"603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905\": rpc error: code = NotFound desc = could not find container \"603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905\": container with ID starting with 603a2f23273447c34d5fbe2b30bb45e9e91e2c3c7ff9adf781ec82ce23941905 not found: ID does not exist" Nov 01 16:32:19.362112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.360429 4911 scope.go:122] "RemoveContainer" containerID="33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3" Nov 01 16:32:19.382110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.382075 4911 scope.go:122] "RemoveContainer" containerID="33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3" Nov 01 16:32:19.382509 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:19.382475 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3\": container with ID starting with 33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3 not found: ID does not exist" containerID="33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3" Nov 01 16:32:19.382615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.382521 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3"} err="failed to get container status \"33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3\": rpc error: code = NotFound desc = could not find container \"33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3\": container with ID starting with 33eb25e1ef83e6f7cab95f476c9874742a84477212ad46258611c728652318d3 not found: ID does not exist" Nov 01 16:32:19.384317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384291 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42x8c\" (UniqueName: \"kubernetes.io/projected/783e58a7-525d-43b7-8da4-0be7c41efd4f-kube-api-access-42x8c\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384318 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384349 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384363 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldlkp\" (UniqueName: \"kubernetes.io/projected/1cc428f9-1f81-49ae-8aee-5f82900fc489-kube-api-access-ldlkp\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384373 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc428f9-1f81-49ae-8aee-5f82900fc489-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384381 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783e58a7-525d-43b7-8da4-0be7c41efd4f-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384390 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384398 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc428f9-1f81-49ae-8aee-5f82900fc489-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384406 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783e58a7-525d-43b7-8da4-0be7c41efd4f-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.384766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.384623 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6"] Nov 01 16:32:19.388674 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.388650 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/premium-w6-43fac8-kserve-6b4bfb4b89-cr6m6"] Nov 01 16:32:19.397843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.397818 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh"] Nov 01 16:32:19.401967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:19.401946 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/simulator-w6-43fac8-kserve-5c9fd5996d-xx6rh"] Nov 01 16:32:20.019083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:20.013600 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" path="/var/lib/kubelet/pods/1cc428f9-1f81-49ae-8aee-5f82900fc489/volumes" Nov 01 16:32:20.019083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:20.016412 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" path="/var/lib/kubelet/pods/783e58a7-525d-43b7-8da4-0be7c41efd4f/volumes" Nov 01 16:32:23.247623 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:23.247582 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" podUID="3e88ab70-e83d-4abd-95a9-96308c0382d1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:23.623647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:23.623557 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2"] Nov 01 16:32:23.623832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:23.623774 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" containerID="cri-o://b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06" gracePeriod=30 Nov 01 16:32:25.010882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.010845 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk"] Nov 01 16:32:25.011409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.011171 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" podUID="663b1737-410d-414f-8e86-c50c45268313" containerName="maas-api" containerID="cri-o://6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da" gracePeriod=30 Nov 01 16:32:25.351324 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.351282 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:32:25.482054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.481531 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663b1737-410d-414f-8e86-c50c45268313-kube-api-access-kpvq6\" (UniqueName: \"kubernetes.io/projected/663b1737-410d-414f-8e86-c50c45268313-kube-api-access-kpvq6\") pod \"663b1737-410d-414f-8e86-c50c45268313\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " Nov 01 16:32:25.482054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.481612 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-maas-api-tls\") pod \"663b1737-410d-414f-8e86-c50c45268313\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " Nov 01 16:32:25.483225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.483189 4911 generic.go:356] "Generic (PLEG): container finished" podID="663b1737-410d-414f-8e86-c50c45268313" containerID="6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da" exitCode=0 Nov 01 16:32:25.483375 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.483352 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" event={"ID":"663b1737-410d-414f-8e86-c50c45268313","Type":"ContainerDied","Data":"6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da"} Nov 01 16:32:25.483450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.483411 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" event={"ID":"663b1737-410d-414f-8e86-c50c45268313","Type":"ContainerDied","Data":"642fbe0d3ff9152bdfbb60987b3f3fc534c2142345288f08c1834e5e399221dc"} Nov 01 16:32:25.483450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.483441 4911 scope.go:122] "RemoveContainer" containerID="6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da" Nov 01 16:32:25.483999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.483622 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk" Nov 01 16:32:25.487064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.486382 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663b1737-410d-414f-8e86-c50c45268313-kube-api-access-kpvq6" pod "663b1737-410d-414f-8e86-c50c45268313" (UID: "663b1737-410d-414f-8e86-c50c45268313"). InnerVolumeSpecName "kube-api-access-kpvq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:25.493188 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.493158 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-maas-api-tls" pod "663b1737-410d-414f-8e86-c50c45268313" (UID: "663b1737-410d-414f-8e86-c50c45268313"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:25.574815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.573012 4911 scope.go:122] "RemoveContainer" containerID="6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da" Nov 01 16:32:25.574815 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:25.573483 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da\": container with ID starting with 6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da not found: ID does not exist" containerID="6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da" Nov 01 16:32:25.574815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.573520 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da"} err="failed to get container status \"6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da\": rpc error: code = NotFound desc = could not find container \"6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da\": container with ID starting with 6034c7ad995336c9bd129136f06591e04c0e2a96862ef5e3dcc3ddc0fb55a3da not found: ID does not exist" Nov 01 16:32:25.582811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.582777 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-metrics-certs\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-metrics-certs\") pod \"663b1737-410d-414f-8e86-c50c45268313\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " Nov 01 16:32:25.583318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.583287 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663b1737-410d-414f-8e86-c50c45268313-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b1737-410d-414f-8e86-c50c45268313-openshift-service-ca\") pod \"663b1737-410d-414f-8e86-c50c45268313\" (UID: \"663b1737-410d-414f-8e86-c50c45268313\") " Nov 01 16:32:25.583727 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.583691 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663b1737-410d-414f-8e86-c50c45268313-openshift-service-ca" pod "663b1737-410d-414f-8e86-c50c45268313" (UID: "663b1737-410d-414f-8e86-c50c45268313"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:25.583880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.583860 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b1737-410d-414f-8e86-c50c45268313-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:25.583932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.583887 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvq6\" (UniqueName: \"kubernetes.io/projected/663b1737-410d-414f-8e86-c50c45268313-kube-api-access-kpvq6\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:25.583932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.583902 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:25.585259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.585227 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-metrics-certs" pod "663b1737-410d-414f-8e86-c50c45268313" (UID: "663b1737-410d-414f-8e86-c50c45268313"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:25.684892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.684843 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663b1737-410d-414f-8e86-c50c45268313-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:25.809514 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.809480 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk"] Nov 01 16:32:25.814425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.814393 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f47890-59947446f5-fkfvk"] Nov 01 16:32:25.972225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:25.972144 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663b1737-410d-414f-8e86-c50c45268313" path="/var/lib/kubelet/pods/663b1737-410d-414f-8e86-c50c45268313/volumes" Nov 01 16:32:28.550645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:28.550587 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4"] Nov 01 16:32:28.965414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:28.965385 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:32:29.075807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.075728 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-envoy\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.075989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.075834 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-data\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.075989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.075861 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-kube-api-access-db99h\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-kube-api-access-db99h\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.075989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.075887 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-token\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-token\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.075989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.075913 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a01b32b-d146-4929-9465-27b0aa46a09c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a01b32b-d146-4929-9465-27b0aa46a09c-istiod-ca-cert\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.076286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076061 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-certs\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.076286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076146 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-socket\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.076286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076200 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-data" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.076286 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076239 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-credential-socket\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.076517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076405 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-podinfo\") pod \"9a01b32b-d146-4929-9465-27b0aa46a09c\" (UID: \"9a01b32b-d146-4929-9465-27b0aa46a09c\") " Nov 01 16:32:29.076517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076362 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a01b32b-d146-4929-9465-27b0aa46a09c-istiod-ca-cert" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:29.076517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076433 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-socket" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.076692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076516 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-credential-socket" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.076692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.076650 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-certs" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.077133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.077099 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.077133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.077123 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a01b32b-d146-4929-9465-27b0aa46a09c-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.077344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.077137 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.077344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.077152 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.077344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.077165 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.078611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.078581 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-kube-api-access-db99h" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "kube-api-access-db99h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.078611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.078601 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-envoy" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.078611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.078587 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-podinfo" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:29.078784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.078613 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-token" pod "9a01b32b-d146-4929-9465-27b0aa46a09c" (UID: "9a01b32b-d146-4929-9465-27b0aa46a09c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.179053 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.179012 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.179241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.179065 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.179241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.179081 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db99h\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-kube-api-access-db99h\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.179241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.179097 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a01b32b-d146-4929-9465-27b0aa46a09c-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.519053 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.519006 4911 generic.go:356] "Generic (PLEG): container finished" podID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerID="b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06" exitCode=0 Nov 01 16:32:29.519216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.519070 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" event={"ID":"9a01b32b-d146-4929-9465-27b0aa46a09c","Type":"ContainerDied","Data":"b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06"} Nov 01 16:32:29.519216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.519090 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" Nov 01 16:32:29.519216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.519131 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2" event={"ID":"9a01b32b-d146-4929-9465-27b0aa46a09c","Type":"ContainerDied","Data":"481b3da01a3f8ff157c0458aa790b8134dde65d6693f57b77fbe5e9a03f82a0d"} Nov 01 16:32:29.519216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.519157 4911 scope.go:122] "RemoveContainer" containerID="b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06" Nov 01 16:32:29.534278 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.534254 4911 scope.go:122] "RemoveContainer" containerID="b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06" Nov 01 16:32:29.535397 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:29.535370 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06\": container with ID starting with b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06 not found: ID does not exist" containerID="b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06" Nov 01 16:32:29.535492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.535409 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06"} err="failed to get container status \"b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06\": rpc error: code = NotFound desc = could not find container \"b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06\": container with ID starting with b5b9ceb7819dacaa4a64a03b1f7fcddedf010a69e238e8d5e405c87310756a06 not found: ID does not exist" Nov 01 16:32:29.547698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.547666 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2"] Nov 01 16:32:29.552888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.552860 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-vzqz2"] Nov 01 16:32:29.968591 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:29.968514 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" path="/var/lib/kubelet/pods/9a01b32b-d146-4929-9465-27b0aa46a09c/volumes" Nov 01 16:32:32.840816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:32.840772 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9"] Nov 01 16:32:32.841413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:32.841096 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" containerID="cri-o://62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65" gracePeriod=30 Nov 01 16:32:33.247398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:33.247360 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" podUID="3e88ab70-e83d-4abd-95a9-96308c0382d1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:33.971231 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:33.971188 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4"] Nov 01 16:32:34.921093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:34.921054 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl"] Nov 01 16:32:34.921291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:34.921268 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" containerID="cri-o://0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d" gracePeriod=30 Nov 01 16:32:35.777056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:35.776169 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl"] Nov 01 16:32:35.777056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:35.776424 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" containerID="cri-o://e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045" gracePeriod=30 Nov 01 16:32:37.122883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:37.122845 4911 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:32:37.123372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:37.122911 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:32:38.130058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.130013 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:32:38.207985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.207945 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-kube-api-access-r489f\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-kube-api-access-r489f\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.208228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.208021 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-certs\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.208228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.208163 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istiod-ca-cert\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.208357 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.208258 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-podinfo\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.208442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.208398 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-certs" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.208717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.208648 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-token\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-token\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.208895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.208873 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istiod-ca-cert" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:38.211340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.211186 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-kube-api-access-r489f" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "kube-api-access-r489f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.211340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.211250 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-token" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.211817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.211791 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-podinfo" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:38.211927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.211808 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-envoy\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.212005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.211925 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-credential-socket\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.212005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.211979 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-data\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.212182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212029 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-socket\") pod \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\" (UID: \"6eaeb7c7-bba4-4e11-83aa-25832f3f5797\") " Nov 01 16:32:38.212265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212240 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-credential-socket" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.212457 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212434 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-data" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.212645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212623 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.212724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212652 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.212724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212668 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.212724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212681 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.212724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212695 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.212724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212709 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r489f\" (UniqueName: \"kubernetes.io/projected/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-kube-api-access-r489f\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.212724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212625 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-socket" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.213109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.212724 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.220557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.220526 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-envoy" pod "6eaeb7c7-bba4-4e11-83aa-25832f3f5797" (UID: "6eaeb7c7-bba4-4e11-83aa-25832f3f5797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.314120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.314028 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.314120 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.314111 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaeb7c7-bba4-4e11-83aa-25832f3f5797-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.600093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.600030 4911 generic.go:356] "Generic (PLEG): container finished" podID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerID="62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65" exitCode=0 Nov 01 16:32:38.600281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.600245 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" event={"ID":"6eaeb7c7-bba4-4e11-83aa-25832f3f5797","Type":"ContainerDied","Data":"62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65"} Nov 01 16:32:38.600331 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.600302 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" event={"ID":"6eaeb7c7-bba4-4e11-83aa-25832f3f5797","Type":"ContainerDied","Data":"818cf3a35990dab327d521c42ee19182b0d8f2b7b8af21fdb598a6b146aeef47"} Nov 01 16:32:38.600383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.600334 4911 scope.go:122] "RemoveContainer" containerID="62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65" Nov 01 16:32:38.600477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.600449 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9" Nov 01 16:32:38.616104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.616080 4911 scope.go:122] "RemoveContainer" containerID="62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65" Nov 01 16:32:38.616491 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:38.616461 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65\": container with ID starting with 62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65 not found: ID does not exist" containerID="62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65" Nov 01 16:32:38.616554 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.616505 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65"} err="failed to get container status \"62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65\": rpc error: code = NotFound desc = could not find container \"62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65\": container with ID starting with 62f7a20daf914d435b4747899f07d9870adcd4a8d11eb33790ee2351b652cb65 not found: ID does not exist" Nov 01 16:32:38.631495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.631465 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9"] Nov 01 16:32:38.638166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:38.638141 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-h5vn9"] Nov 01 16:32:39.076429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.076403 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:32:39.236832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.235820 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e88ab70-e83d-4abd-95a9-96308c0382d1-kube-api-access-rdfdv\" (UniqueName: \"kubernetes.io/projected/3e88ab70-e83d-4abd-95a9-96308c0382d1-kube-api-access-rdfdv\") pod \"3e88ab70-e83d-4abd-95a9-96308c0382d1\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " Nov 01 16:32:39.236832 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.236009 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e88ab70-e83d-4abd-95a9-96308c0382d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e88ab70-e83d-4abd-95a9-96308c0382d1-plugins-config-volume\") pod \"3e88ab70-e83d-4abd-95a9-96308c0382d1\" (UID: \"3e88ab70-e83d-4abd-95a9-96308c0382d1\") " Nov 01 16:32:39.260406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.238964 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e88ab70-e83d-4abd-95a9-96308c0382d1-plugins-config-volume" pod "3e88ab70-e83d-4abd-95a9-96308c0382d1" (UID: "3e88ab70-e83d-4abd-95a9-96308c0382d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:39.260406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.242967 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e88ab70-e83d-4abd-95a9-96308c0382d1-kube-api-access-rdfdv" pod "3e88ab70-e83d-4abd-95a9-96308c0382d1" (UID: "3e88ab70-e83d-4abd-95a9-96308c0382d1"). InnerVolumeSpecName "kube-api-access-rdfdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:39.338029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.337997 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdfdv\" (UniqueName: \"kubernetes.io/projected/3e88ab70-e83d-4abd-95a9-96308c0382d1-kube-api-access-rdfdv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:39.338029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.338026 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e88ab70-e83d-4abd-95a9-96308c0382d1-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:39.610741 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.610660 4911 generic.go:356] "Generic (PLEG): container finished" podID="3e88ab70-e83d-4abd-95a9-96308c0382d1" containerID="d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51" exitCode=0 Nov 01 16:32:39.610741 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.610708 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" Nov 01 16:32:39.610920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.610742 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" event={"ID":"3e88ab70-e83d-4abd-95a9-96308c0382d1","Type":"ContainerDied","Data":"d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51"} Nov 01 16:32:39.610920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.610802 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926" event={"ID":"3e88ab70-e83d-4abd-95a9-96308c0382d1","Type":"ContainerDied","Data":"bbbd124951879ca07765d998215efb34f0fae607241573239f693fc2a8d6e9f6"} Nov 01 16:32:39.610920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.610826 4911 scope.go:122] "RemoveContainer" containerID="d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51" Nov 01 16:32:39.624553 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.624528 4911 scope.go:122] "RemoveContainer" containerID="d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51" Nov 01 16:32:39.624896 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:39.624872 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51\": container with ID starting with d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51 not found: ID does not exist" containerID="d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51" Nov 01 16:32:39.624949 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.624907 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51"} err="failed to get container status \"d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51\": rpc error: code = NotFound desc = could not find container \"d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51\": container with ID starting with d3d3a956a53fed9b3b529049a2ab940a52927303312bd8f2a952dac228de1e51 not found: ID does not exist" Nov 01 16:32:39.642773 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.642741 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926"] Nov 01 16:32:39.649402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.649370 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f47890-b7b766d4c-bm926"] Nov 01 16:32:39.987516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.987479 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e88ab70-e83d-4abd-95a9-96308c0382d1" path="/var/lib/kubelet/pods/3e88ab70-e83d-4abd-95a9-96308c0382d1/volumes" Nov 01 16:32:39.989932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:39.988175 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" path="/var/lib/kubelet/pods/6eaeb7c7-bba4-4e11-83aa-25832f3f5797/volumes" Nov 01 16:32:40.014650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.014601 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hfzvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 01 16:32:40.014824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.014696 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 01 16:32:40.245112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.245084 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 16:32:40.265661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.265631 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-kube-api-access-8vxlw\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-kube-api-access-8vxlw\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.265814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.265683 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-podinfo\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.265814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.265770 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-socket\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.265953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.265863 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-token\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-token\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.266007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.265939 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istiod-ca-cert\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.266081 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266010 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-credential-socket\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.266181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266081 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-socket" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:40.266181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266119 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-envoy\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.266329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266270 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-credential-socket" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:40.266387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266353 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istiod-ca-cert" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:40.266821 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266799 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.266906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266825 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.266906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.266841 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.268303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.268279 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-kube-api-access-8vxlw" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "kube-api-access-8vxlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:40.268408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.268363 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-envoy" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:40.268535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.268515 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-podinfo" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:40.268743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.268721 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-token" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:40.367340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.367305 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-certs\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.367340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.367354 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-data\") pod \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\" (UID: \"68c8d771-57c0-4eec-9062-c2cf60a3aa79\") " Nov 01 16:32:40.367683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.367656 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-certs" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:40.367759 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.367701 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-data" pod "68c8d771-57c0-4eec-9062-c2cf60a3aa79" (UID: "68c8d771-57c0-4eec-9062-c2cf60a3aa79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:40.368080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.368027 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.368197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.368085 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.368197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.368095 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.368197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.368104 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.368197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.368117 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxlw\" (UniqueName: \"kubernetes.io/projected/68c8d771-57c0-4eec-9062-c2cf60a3aa79-kube-api-access-8vxlw\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.368197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.368127 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c8d771-57c0-4eec-9062-c2cf60a3aa79-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.623957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.623910 4911 generic.go:356] "Generic (PLEG): container finished" podID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerID="0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d" exitCode=0 Nov 01 16:32:40.624547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.624499 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" event={"ID":"68c8d771-57c0-4eec-9062-c2cf60a3aa79","Type":"ContainerDied","Data":"0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d"} Nov 01 16:32:40.624859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.624834 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" event={"ID":"68c8d771-57c0-4eec-9062-c2cf60a3aa79","Type":"ContainerDied","Data":"34f1bb32c66e077287d1788f41af09dc5d277afc960292f541dab47d30f0752c"} Nov 01 16:32:40.625031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.625012 4911 scope.go:122] "RemoveContainer" containerID="0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d" Nov 01 16:32:40.625353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.625334 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl" Nov 01 16:32:40.642735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.642713 4911 scope.go:122] "RemoveContainer" containerID="0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d" Nov 01 16:32:40.643023 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:40.642998 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d\": container with ID starting with 0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d not found: ID does not exist" containerID="0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d" Nov 01 16:32:40.643102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.643031 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d"} err="failed to get container status \"0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d\": rpc error: code = NotFound desc = could not find container \"0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d\": container with ID starting with 0d33e6728d26a5a2664a1191e7e99ec23eed0678384ecff2c09328532185cd4d not found: ID does not exist" Nov 01 16:32:40.654640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.654612 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl"] Nov 01 16:32:40.658724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.658699 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hfzvl"] Nov 01 16:32:40.758862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.758835 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:32:40.766403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:40.766378 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:32:40.897458 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:40.897372 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:32:41.150082 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.149989 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:32:41.179102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179060 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-podinfo\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179122 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-data\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179173 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-token\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179299 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-envoy\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179354 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-socket\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179393 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-kube-api-access-rz52z\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-kube-api-access-rz52z\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179445 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-credential-socket\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179528 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-certs\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179601 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbf1230-7fdf-4775-b710-758043bf0a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf1230-7fdf-4775-b710-758043bf0a79-istiod-ca-cert\") pod \"bbbf1230-7fdf-4775-b710-758043bf0a79\" (UID: \"bbbf1230-7fdf-4775-b710-758043bf0a79\") " Nov 01 16:32:41.179785 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179675 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-socket" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.179974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.179928 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-credential-socket" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.180326 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.180304 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.180414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.180337 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.180414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.180303 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-certs" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.180414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.180314 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf1230-7fdf-4775-b710-758043bf0a79-istiod-ca-cert" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:41.180414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.180320 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-data" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.184446 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.184384 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-kube-api-access-rz52z" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "kube-api-access-rz52z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:41.184446 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.184422 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-token" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:41.184765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.184709 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-envoy" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:41.185927 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.185903 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-podinfo" pod "bbbf1230-7fdf-4775-b710-758043bf0a79" (UID: "bbbf1230-7fdf-4775-b710-758043bf0a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:41.281417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.281375 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz52z\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-kube-api-access-rz52z\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.281417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.281412 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.281417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.281422 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf1230-7fdf-4775-b710-758043bf0a79-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.282060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.281430 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.282060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.281439 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.282060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.281448 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.282060 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.281456 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf1230-7fdf-4775-b710-758043bf0a79-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:41.642221 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.642184 4911 generic.go:356] "Generic (PLEG): container finished" podID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerID="e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045" exitCode=0 Nov 01 16:32:41.642415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.642236 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" Nov 01 16:32:41.642415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.642275 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" event={"ID":"bbbf1230-7fdf-4775-b710-758043bf0a79","Type":"ContainerDied","Data":"e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045"} Nov 01 16:32:41.642415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.642338 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl" event={"ID":"bbbf1230-7fdf-4775-b710-758043bf0a79","Type":"ContainerDied","Data":"83c2d6d8bf5e47856a6a626c4424465da5a285251cb6e5d06c8145c84a5b138a"} Nov 01 16:32:41.642415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.642368 4911 scope.go:122] "RemoveContainer" containerID="e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045" Nov 01 16:32:41.658574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.658545 4911 scope.go:122] "RemoveContainer" containerID="e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045" Nov 01 16:32:41.662492 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:41.662452 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045\": container with ID starting with e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045 not found: ID does not exist" containerID="e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045" Nov 01 16:32:41.662620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.662503 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045"} err="failed to get container status \"e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045\": rpc error: code = NotFound desc = could not find container \"e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045\": container with ID starting with e94618e6207e545b8a2d6c1711633cad683d440a00bf661ec1f49ac1a95c9045 not found: ID does not exist" Nov 01 16:32:41.670183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.670153 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl"] Nov 01 16:32:41.673869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.673838 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-m7cfl"] Nov 01 16:32:41.968705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.968609 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" path="/var/lib/kubelet/pods/68c8d771-57c0-4eec-9062-c2cf60a3aa79/volumes" Nov 01 16:32:41.969305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:41.969285 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" path="/var/lib/kubelet/pods/bbbf1230-7fdf-4775-b710-758043bf0a79/volumes" Nov 01 16:32:43.577342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:43.577299 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" podUID="9811d4e5-fe1b-4920-9601-fc298fb4ab66" containerName="main" containerID="cri-o://aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02" gracePeriod=45 Nov 01 16:32:43.938495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:43.938462 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:32:44.048683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.048643 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9811d4e5-fe1b-4920-9601-fc298fb4ab66-kube-api-access-nfkcb\" (UniqueName: \"kubernetes.io/projected/9811d4e5-fe1b-4920-9601-fc298fb4ab66-kube-api-access-nfkcb\") pod \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " Nov 01 16:32:44.048865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.048787 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-home\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-home\") pod \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " Nov 01 16:32:44.048969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.048951 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tls-certs\" (UniqueName: \"kubernetes.io/secret/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tls-certs\") pod \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " Nov 01 16:32:44.049012 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.048998 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tmp-dir\") pod \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " Nov 01 16:32:44.049114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049031 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-dshm\") pod \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " Nov 01 16:32:44.049114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049058 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-home" pod "9811d4e5-fe1b-4920-9601-fc298fb4ab66" (UID: "9811d4e5-fe1b-4920-9601-fc298fb4ab66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:44.049239 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049129 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-model-cache\") pod \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\" (UID: \"9811d4e5-fe1b-4920-9601-fc298fb4ab66\") " Nov 01 16:32:44.049317 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049290 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tmp-dir" pod "9811d4e5-fe1b-4920-9601-fc298fb4ab66" (UID: "9811d4e5-fe1b-4920-9601-fc298fb4ab66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:44.049443 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049417 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-model-cache" pod "9811d4e5-fe1b-4920-9601-fc298fb4ab66" (UID: "9811d4e5-fe1b-4920-9601-fc298fb4ab66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:44.049929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049881 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:44.049929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049913 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:44.050093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.049929 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:44.051402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.051379 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9811d4e5-fe1b-4920-9601-fc298fb4ab66-kube-api-access-nfkcb" pod "9811d4e5-fe1b-4920-9601-fc298fb4ab66" (UID: "9811d4e5-fe1b-4920-9601-fc298fb4ab66"). InnerVolumeSpecName "kube-api-access-nfkcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:44.051499 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.051482 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-dshm" pod "9811d4e5-fe1b-4920-9601-fc298fb4ab66" (UID: "9811d4e5-fe1b-4920-9601-fc298fb4ab66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:44.051611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.051598 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tls-certs" pod "9811d4e5-fe1b-4920-9601-fc298fb4ab66" (UID: "9811d4e5-fe1b-4920-9601-fc298fb4ab66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:44.151899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.151805 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9811d4e5-fe1b-4920-9601-fc298fb4ab66-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:44.151899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.151849 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9811d4e5-fe1b-4920-9601-fc298fb4ab66-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:44.151899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.151866 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfkcb\" (UniqueName: \"kubernetes.io/projected/9811d4e5-fe1b-4920-9601-fc298fb4ab66-kube-api-access-nfkcb\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:44.704925 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.704889 4911 generic.go:356] "Generic (PLEG): container finished" podID="9811d4e5-fe1b-4920-9601-fc298fb4ab66" containerID="aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02" exitCode=0 Nov 01 16:32:44.705414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.704935 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" Nov 01 16:32:44.705414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.704976 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" event={"ID":"9811d4e5-fe1b-4920-9601-fc298fb4ab66","Type":"ContainerDied","Data":"aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02"} Nov 01 16:32:44.705414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.705026 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4" event={"ID":"9811d4e5-fe1b-4920-9601-fc298fb4ab66","Type":"ContainerDied","Data":"a9ec1f8055a50371ce18155dd22561986f828c81b341ecce593cf93bd58c0876"} Nov 01 16:32:44.705414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.705087 4911 scope.go:122] "RemoveContainer" containerID="aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02" Nov 01 16:32:44.716344 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.716323 4911 scope.go:122] "RemoveContainer" containerID="aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02" Nov 01 16:32:44.716671 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:44.716649 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02\": container with ID starting with aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02 not found: ID does not exist" containerID="aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02" Nov 01 16:32:44.716754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.716677 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02"} err="failed to get container status \"aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02\": rpc error: code = NotFound desc = could not find container \"aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02\": container with ID starting with aad3f16feb14aa884387f73b37733125c55f2b901d715a5203927a1f86fdfd02 not found: ID does not exist" Nov 01 16:32:44.731104 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.731073 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4"] Nov 01 16:32:44.734239 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:44.734215 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-383fde/sub-test-model-383fde-kserve-57697d44cf-csld4"] Nov 01 16:32:45.967529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:45.967490 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9811d4e5-fe1b-4920-9601-fc298fb4ab66" path="/var/lib/kubelet/pods/9811d4e5-fe1b-4920-9601-fc298fb4ab66/volumes" Nov 01 16:32:46.803777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.803734 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7"] Nov 01 16:32:46.804358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804336 4911 cpu_manager.go:414] "containerMap: removing container" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" Nov 01 16:32:46.804358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804355 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804371 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9811d4e5-fe1b-4920-9601-fc298fb4ab66" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804376 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9811d4e5-fe1b-4920-9601-fc298fb4ab66" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804384 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804389 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804401 4911 cpu_manager.go:414] "containerMap: removing container" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804406 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804419 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804424 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804432 4911 cpu_manager.go:414] "containerMap: removing container" podUID="663b1737-410d-414f-8e86-c50c45268313" containerName="maas-api" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804437 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663b1737-410d-414f-8e86-c50c45268313" containerName="maas-api" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804445 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804450 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804460 4911 cpu_manager.go:414] "containerMap: removing container" podUID="3e88ab70-e83d-4abd-95a9-96308c0382d1" containerName="payload-processing" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804464 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e88ab70-e83d-4abd-95a9-96308c0382d1" containerName="payload-processing" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804486 4911 cpu_manager.go:414] "containerMap: removing container" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804491 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804500 4911 cpu_manager.go:414] "containerMap: removing container" podUID="52753ffd-ccbf-4250-a1dc-989d4dad113a" containerName="payload-pre-processing" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804504 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52753ffd-ccbf-4250-a1dc-989d4dad113a" containerName="payload-pre-processing" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804520 4911 cpu_manager.go:414] "containerMap: removing container" podUID="91743d76-57e1-41b8-af00-9bac5b4ce470" containerName="maas-api" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804527 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91743d76-57e1-41b8-af00-9bac5b4ce470" containerName="maas-api" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804538 4911 cpu_manager.go:414] "containerMap: removing container" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" Nov 01 16:32:46.804560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804544 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804641 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="91743d76-57e1-41b8-af00-9bac5b4ce470" containerName="maas-api" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804651 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbf1230-7fdf-4775-b710-758043bf0a79" containerName="istio-proxy" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804659 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eaeb7c7-bba4-4e11-83aa-25832f3f5797" containerName="istio-proxy" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804667 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9811d4e5-fe1b-4920-9601-fc298fb4ab66" containerName="main" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804673 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b523b317-2c74-4159-a618-ce4a6d9635c5" containerName="main" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804680 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c8d771-57c0-4eec-9062-c2cf60a3aa79" containerName="istio-proxy" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804687 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a01b32b-d146-4929-9465-27b0aa46a09c" containerName="istio-proxy" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804694 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="663b1737-410d-414f-8e86-c50c45268313" containerName="maas-api" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804699 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc428f9-1f81-49ae-8aee-5f82900fc489" containerName="main" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804706 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e88ab70-e83d-4abd-95a9-96308c0382d1" containerName="payload-processing" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804715 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="783e58a7-525d-43b7-8da4-0be7c41efd4f" containerName="main" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804720 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="52753ffd-ccbf-4250-a1dc-989d4dad113a" containerName="payload-pre-processing" Nov 01 16:32:46.805366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.804726 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74269e5-3eee-4c31-acf1-a610e61b8ed3" containerName="istio-proxy" Nov 01 16:32:46.809764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.809739 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.814921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.814407 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-f01de5-openshift-default-dockercfg-czkss\"" Nov 01 16:32:46.820939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.820724 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7"] Nov 01 16:32:46.902468 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902431 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.902667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902501 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.902667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902572 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf9c983-3687-4898-b4bd-744157f4bdd2-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.902667 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902637 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.902847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902726 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fh7v\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-kube-api-access-5fh7v\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.902847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902757 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.902847 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902788 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.902963 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.902937 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:46.903081 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:46.903034 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004340 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004407 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004481 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004511 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004528 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf9c983-3687-4898-b4bd-744157f4bdd2-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004589 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004629 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fh7v\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-kube-api-access-5fh7v\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004653 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004678 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.004935 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.004889 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.005369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.005075 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.005369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.005139 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.005369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.005217 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.005369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.005348 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf9c983-3687-4898-b4bd-744157f4bdd2-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.007161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.007137 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.007161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.007142 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.012507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.012482 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.012807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.012782 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fh7v\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-kube-api-access-5fh7v\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.129100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.128987 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:47.431764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.431730 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7"] Nov 01 16:32:47.433379 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:32:47.433352 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf9c983_3687_4898_b4bd_744157f4bdd2.slice/crio-63af40dabc3414e88cefcfecf7ccbafba974752befeb719c719be51b134441bc WatchSource:0}: Error finding container 63af40dabc3414e88cefcfecf7ccbafba974752befeb719c719be51b134441bc: Status 404 returned error can't find the container with id 63af40dabc3414e88cefcfecf7ccbafba974752befeb719c719be51b134441bc Nov 01 16:32:47.436729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.436693 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:47.436844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.436759 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:47.436844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.436790 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:47.730442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.730336 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" event={"ID":"ccf9c983-3687-4898-b4bd-744157f4bdd2","Type":"ContainerStarted","Data":"67cda1fec47b3d2969ccb95f7a18357a2efc99e8a5023e47dc3a1dcf00e09a6e"} Nov 01 16:32:47.730442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.730403 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" event={"ID":"ccf9c983-3687-4898-b4bd-744157f4bdd2","Type":"ContainerStarted","Data":"63af40dabc3414e88cefcfecf7ccbafba974752befeb719c719be51b134441bc"} Nov 01 16:32:47.756460 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:47.756386 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podStartSLOduration=1.756363508 podStartE2EDuration="1.756363508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:47.753093169 +0000 UTC m=+2707.897547811" watchObservedRunningTime="2026-11-01 16:32:47.756363508 +0000 UTC m=+2707.900818149" Nov 01 16:32:48.135676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:48.135636 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:48.139059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:48.139002 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:32:48.139315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:48.139283 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 16:32:48.993389 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:48.993343 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" podUID="a8a96349-9c37-491b-b324-6bd81ed6f07d" containerName="main" containerID="cri-o://7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79" gracePeriod=45 Nov 01 16:32:49.129957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.129911 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:32:49.130240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.129974 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 16:32:49.439319 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.439272 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:32:49.525497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.525451 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-home\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-home\") pod \"a8a96349-9c37-491b-b324-6bd81ed6f07d\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " Nov 01 16:32:49.525682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.525515 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a96349-9c37-491b-b324-6bd81ed6f07d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a96349-9c37-491b-b324-6bd81ed6f07d-tls-certs\") pod \"a8a96349-9c37-491b-b324-6bd81ed6f07d\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " Nov 01 16:32:49.525682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.525556 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a96349-9c37-491b-b324-6bd81ed6f07d-kube-api-access-zmcr5\" (UniqueName: \"kubernetes.io/projected/a8a96349-9c37-491b-b324-6bd81ed6f07d-kube-api-access-zmcr5\") pod \"a8a96349-9c37-491b-b324-6bd81ed6f07d\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " Nov 01 16:32:49.525682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.525614 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-model-cache\") pod \"a8a96349-9c37-491b-b324-6bd81ed6f07d\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " Nov 01 16:32:49.525882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.525822 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-tmp-dir\") pod \"a8a96349-9c37-491b-b324-6bd81ed6f07d\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " Nov 01 16:32:49.525882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.525855 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-dshm\") pod \"a8a96349-9c37-491b-b324-6bd81ed6f07d\" (UID: \"a8a96349-9c37-491b-b324-6bd81ed6f07d\") " Nov 01 16:32:49.526624 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.526593 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-model-cache" pod "a8a96349-9c37-491b-b324-6bd81ed6f07d" (UID: "a8a96349-9c37-491b-b324-6bd81ed6f07d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:49.526854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.526804 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-tmp-dir" pod "a8a96349-9c37-491b-b324-6bd81ed6f07d" (UID: "a8a96349-9c37-491b-b324-6bd81ed6f07d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:49.527063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.527014 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-home" pod "a8a96349-9c37-491b-b324-6bd81ed6f07d" (UID: "a8a96349-9c37-491b-b324-6bd81ed6f07d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:49.530884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.529198 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-dshm" pod "a8a96349-9c37-491b-b324-6bd81ed6f07d" (UID: "a8a96349-9c37-491b-b324-6bd81ed6f07d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:49.530884 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.530581 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a96349-9c37-491b-b324-6bd81ed6f07d-tls-certs" pod "a8a96349-9c37-491b-b324-6bd81ed6f07d" (UID: "a8a96349-9c37-491b-b324-6bd81ed6f07d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:49.533114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.532533 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a96349-9c37-491b-b324-6bd81ed6f07d-kube-api-access-zmcr5" pod "a8a96349-9c37-491b-b324-6bd81ed6f07d" (UID: "a8a96349-9c37-491b-b324-6bd81ed6f07d"). InnerVolumeSpecName "kube-api-access-zmcr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:49.627645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.627550 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:49.627645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.627590 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:49.627645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.627602 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:49.627645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.627615 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a96349-9c37-491b-b324-6bd81ed6f07d-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:49.627645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.627628 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcr5\" (UniqueName: \"kubernetes.io/projected/a8a96349-9c37-491b-b324-6bd81ed6f07d-kube-api-access-zmcr5\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:49.627645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.627643 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a96349-9c37-491b-b324-6bd81ed6f07d-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:49.758809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.758767 4911 generic.go:356] "Generic (PLEG): container finished" podID="a8a96349-9c37-491b-b324-6bd81ed6f07d" containerID="7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79" exitCode=0 Nov 01 16:32:49.760187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.760160 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" Nov 01 16:32:49.789067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.775103 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" event={"ID":"a8a96349-9c37-491b-b324-6bd81ed6f07d","Type":"ContainerDied","Data":"7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79"} Nov 01 16:32:49.789067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.775183 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4" event={"ID":"a8a96349-9c37-491b-b324-6bd81ed6f07d","Type":"ContainerDied","Data":"6894e28fd2b4cf26443265ab507863dca55a8e313be3bac555c8266a588cac83"} Nov 01 16:32:49.789067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.775210 4911 scope.go:122] "RemoveContainer" containerID="7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79" Nov 01 16:32:49.802542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.801860 4911 scope.go:122] "RemoveContainer" containerID="7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79" Nov 01 16:32:49.802542 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:49.802326 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79\": container with ID starting with 7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79 not found: ID does not exist" containerID="7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79" Nov 01 16:32:49.802542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.802376 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79"} err="failed to get container status \"7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79\": rpc error: code = NotFound desc = could not find container \"7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79\": container with ID starting with 7a81717c795336c008b4fcf77b367947f4ae168ed518dbd368a0267366b96e79 not found: ID does not exist" Nov 01 16:32:49.812402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.812366 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4"] Nov 01 16:32:49.819734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.819701 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a9a967/sub-test-model-a9a967-kserve-864487d8bc-qp9g4"] Nov 01 16:32:49.988536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:49.988497 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a96349-9c37-491b-b324-6bd81ed6f07d" path="/var/lib/kubelet/pods/a8a96349-9c37-491b-b324-6bd81ed6f07d/volumes" Nov 01 16:32:50.136259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:50.136221 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:32:50.136513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:50.136483 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 16:32:51.129567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:51.129529 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:32:51.130004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:51.129597 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 16:32:52.096865 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.096808 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65"] Nov 01 16:32:52.097284 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.097250 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" podUID="de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" containerName="maas-api" containerID="cri-o://4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd" gracePeriod=30 Nov 01 16:32:52.138254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.136549 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:32:52.138254 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.136602 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 16:32:52.707985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.707529 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:32:52.805441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.805404 4911 generic.go:356] "Generic (PLEG): container finished" podID="de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" containerID="4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd" exitCode=0 Nov 01 16:32:52.805639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.805474 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" Nov 01 16:32:52.805639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.805603 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" event={"ID":"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47","Type":"ContainerDied","Data":"4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd"} Nov 01 16:32:52.805764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.805686 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65" event={"ID":"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47","Type":"ContainerDied","Data":"5d5920fce0d28025b0a7c253829a70600a53847f1a3558ac48708242a001eb27"} Nov 01 16:32:52.805764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.805719 4911 scope.go:122] "RemoveContainer" containerID="4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd" Nov 01 16:32:52.818787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.818763 4911 scope.go:122] "RemoveContainer" containerID="4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd" Nov 01 16:32:52.819110 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:32:52.819086 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd\": container with ID starting with 4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd not found: ID does not exist" containerID="4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd" Nov 01 16:32:52.819203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.819132 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd"} err="failed to get container status \"4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd\": rpc error: code = NotFound desc = could not find container \"4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd\": container with ID starting with 4a576314241794116b278f304bf2ea2de81ec61693cfc09b67e632f58b63eadd not found: ID does not exist" Nov 01 16:32:52.898197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.895737 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-openshift-service-ca\") pod \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " Nov 01 16:32:52.898197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.895891 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-metrics-certs\") pod \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " Nov 01 16:32:52.898197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.896089 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-kube-api-access-75m8k\" (UniqueName: \"kubernetes.io/projected/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-kube-api-access-75m8k\") pod \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " Nov 01 16:32:52.898197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.896125 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls\") pod \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\" (UID: \"de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47\") " Nov 01 16:32:52.898197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.896672 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-openshift-service-ca" pod "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" (UID: "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:52.898197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.896891 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:52.898951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.898882 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls" pod "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" (UID: "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:52.903078 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.902189 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-metrics-certs" pod "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" (UID: "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:52.903078 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.902246 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-kube-api-access-75m8k" pod "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" (UID: "de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47"). InnerVolumeSpecName "kube-api-access-75m8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:52.998538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.998501 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75m8k\" (UniqueName: \"kubernetes.io/projected/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-kube-api-access-75m8k\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:52.998538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.998535 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:52.998538 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:52.998545 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:53.129316 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:53.129285 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:32:53.129459 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:53.129347 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 16:32:53.132493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:53.132463 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65"] Nov 01 16:32:53.138061 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:53.138020 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-734859-568d79c658-l5v65"] Nov 01 16:32:53.966911 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:53.966873 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" path="/var/lib/kubelet/pods/de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47/volumes" Nov 01 16:32:54.220138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:54.217182 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:54.220138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:54.218019 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:54.220138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:54.219004 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:32:54.577289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:54.577253 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh"] Nov 01 16:32:59.788271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:59.788235 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m"] Nov 01 16:32:59.788691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:32:59.788442 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" containerID="cri-o://b348296c028eb62561e4c632dd256474ccb931b0aa03068645645e676888fff6" gracePeriod=30 Nov 01 16:33:00.598103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:00.598031 4911 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:33:00.598289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:00.598140 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:33:04.962423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:04.962389 4911 generic.go:356] "Generic (PLEG): container finished" podID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerID="b348296c028eb62561e4c632dd256474ccb931b0aa03068645645e676888fff6" exitCode=0 Nov 01 16:33:04.963007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:04.962457 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" event={"ID":"b4ecd3e2-ea3f-461e-b436-36910fc14f95","Type":"ContainerDied","Data":"b348296c028eb62561e4c632dd256474ccb931b0aa03068645645e676888fff6"} Nov 01 16:33:05.124834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.113892 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:33:05.227654 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.227615 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-certs\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.227971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.227953 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-kube-api-access-w5bmc\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-kube-api-access-w5bmc\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.228111 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.228092 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-podinfo\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.228250 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.228237 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istiod-ca-cert\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.228381 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.228368 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-data\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.228540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.228525 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-credential-socket\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.228716 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.228701 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-token\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-token\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.228886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.228869 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-socket\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.229356 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.229337 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-envoy\") pod \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\" (UID: \"b4ecd3e2-ea3f-461e-b436-36910fc14f95\") " Nov 01 16:33:05.231647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.228886 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-credential-socket" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.236564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.232889 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-kube-api-access-w5bmc" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "kube-api-access-w5bmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:05.236564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.233365 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-data" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.236564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.235365 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istiod-ca-cert" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:05.241132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.238657 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l"] Nov 01 16:33:05.241132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.239620 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-socket" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.241132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.240654 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-certs" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.269360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.269318 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-podinfo" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:05.269861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.269821 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-envoy" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.274230 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.274203 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-token" pod "b4ecd3e2-ea3f-461e-b436-36910fc14f95" (UID: "b4ecd3e2-ea3f-461e-b436-36910fc14f95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:05.331272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331235 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5bmc\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-kube-api-access-w5bmc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331268 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331272 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331278 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331287 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331297 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331305 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331314 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331322 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.331565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.331330 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ecd3e2-ea3f-461e-b436-36910fc14f95-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.971663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.971630 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" Nov 01 16:33:05.972189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.971652 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m" event={"ID":"b4ecd3e2-ea3f-461e-b436-36910fc14f95","Type":"ContainerDied","Data":"afc8ba2bb46a0c8484aa64e6c0dd937851cea205d850eb92c050f8ee41550e72"} Nov 01 16:33:05.972189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:05.971731 4911 scope.go:122] "RemoveContainer" containerID="b348296c028eb62561e4c632dd256474ccb931b0aa03068645645e676888fff6" Nov 01 16:33:06.005361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:06.005331 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m"] Nov 01 16:33:06.010615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:06.010591 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-lkr5m"] Nov 01 16:33:07.968664 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:07.968628 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" path="/var/lib/kubelet/pods/b4ecd3e2-ea3f-461e-b436-36910fc14f95/volumes" Nov 01 16:33:09.599820 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.599774 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" podUID="c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" containerName="main" containerID="cri-o://8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0" gracePeriod=45 Nov 01 16:33:09.878269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.878235 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:33:09.930325 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.930287 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-dshm\") pod \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " Nov 01 16:33:09.930610 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.930586 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tmp-dir\") pod \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " Nov 01 16:33:09.930828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.930799 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-kube-api-access-mslg4\" (UniqueName: \"kubernetes.io/projected/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-kube-api-access-mslg4\") pod \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " Nov 01 16:33:09.931535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.931220 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tmp-dir" pod "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" (UID: "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:09.931697 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.931642 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-home\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-home\") pod \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " Nov 01 16:33:09.931887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.931860 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-home" pod "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" (UID: "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:09.932101 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.932077 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-model-cache\") pod \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " Nov 01 16:33:09.932243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.932226 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tls-certs\" (UniqueName: \"kubernetes.io/secret/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tls-certs\") pod \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\" (UID: \"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5\") " Nov 01 16:33:09.932886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.932862 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.933014 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.932998 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:09.933315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.933296 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-model-cache" pod "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" (UID: "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:09.939503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.939470 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-kube-api-access-mslg4" pod "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" (UID: "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5"). InnerVolumeSpecName "kube-api-access-mslg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:09.940630 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.940594 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-dshm" pod "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" (UID: "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:09.941633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:09.941610 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tls-certs" pod "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" (UID: "c50ef3d7-ab6e-4009-8a8d-78d80de3ead5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:10.009883 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.009843 4911 generic.go:356] "Generic (PLEG): container finished" podID="c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" containerID="8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0" exitCode=0 Nov 01 16:33:10.010115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.009886 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" Nov 01 16:33:10.010115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.009902 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" event={"ID":"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5","Type":"ContainerDied","Data":"8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0"} Nov 01 16:33:10.010115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.009940 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh" event={"ID":"c50ef3d7-ab6e-4009-8a8d-78d80de3ead5","Type":"ContainerDied","Data":"a97733fecbdb322e76209cbc5fe42efbf4877cb27e59fa8f72bbb89246b0221f"} Nov 01 16:33:10.010115 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.009958 4911 scope.go:122] "RemoveContainer" containerID="8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0" Nov 01 16:33:10.022676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.022649 4911 scope.go:122] "RemoveContainer" containerID="8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0" Nov 01 16:33:10.022999 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:10.022970 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0\": container with ID starting with 8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0 not found: ID does not exist" containerID="8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0" Nov 01 16:33:10.023099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.023000 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0"} err="failed to get container status \"8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0\": rpc error: code = NotFound desc = could not find container \"8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0\": container with ID starting with 8745bb738fa4ab6b9e4cc3db9c2ff101e152e232e085ee25a1c517e113bf6cf0 not found: ID does not exist" Nov 01 16:33:10.030077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.030052 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh"] Nov 01 16:33:10.034723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.034693 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:10.034843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.034727 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mslg4\" (UniqueName: \"kubernetes.io/projected/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-kube-api-access-mslg4\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:10.034843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.034745 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:10.034843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.034760 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:10.035149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:10.035127 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-f01de5/rate-test-model-f01de5-kserve-6bb95857df-8sdhh"] Nov 01 16:33:10.871050 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:10.870992 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:33:11.967208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:11.967175 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" path="/var/lib/kubelet/pods/c50ef3d7-ab6e-4009-8a8d-78d80de3ead5/volumes" Nov 01 16:33:13.658368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:13.658330 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx"] Nov 01 16:33:13.658881 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:13.658579 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" podUID="ead98060-0354-4a0d-b90c-92ee70db832d" containerName="maas-api" containerID="cri-o://2c5cb48a550ce4d9e7a72328476ebaf79af2134766a4b982bcbcb2b0d3627927" gracePeriod=30 Nov 01 16:33:14.082523 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.082489 4911 generic.go:356] "Generic (PLEG): container finished" podID="ead98060-0354-4a0d-b90c-92ee70db832d" containerID="2c5cb48a550ce4d9e7a72328476ebaf79af2134766a4b982bcbcb2b0d3627927" exitCode=0 Nov 01 16:33:14.082692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.082678 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" event={"ID":"ead98060-0354-4a0d-b90c-92ee70db832d","Type":"ContainerDied","Data":"2c5cb48a550ce4d9e7a72328476ebaf79af2134766a4b982bcbcb2b0d3627927"} Nov 01 16:33:14.196544 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.196471 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:33:14.234966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.234925 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead98060-0354-4a0d-b90c-92ee70db832d-kube-api-access-vsd9r\" (UniqueName: \"kubernetes.io/projected/ead98060-0354-4a0d-b90c-92ee70db832d-kube-api-access-vsd9r\") pod \"ead98060-0354-4a0d-b90c-92ee70db832d\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " Nov 01 16:33:14.235219 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.235018 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead98060-0354-4a0d-b90c-92ee70db832d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead98060-0354-4a0d-b90c-92ee70db832d-openshift-service-ca\") pod \"ead98060-0354-4a0d-b90c-92ee70db832d\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " Nov 01 16:33:14.235219 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.235101 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-metrics-certs\") pod \"ead98060-0354-4a0d-b90c-92ee70db832d\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " Nov 01 16:33:14.235379 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.235224 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls\") pod \"ead98060-0354-4a0d-b90c-92ee70db832d\" (UID: \"ead98060-0354-4a0d-b90c-92ee70db832d\") " Nov 01 16:33:14.235502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.235470 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead98060-0354-4a0d-b90c-92ee70db832d-openshift-service-ca" pod "ead98060-0354-4a0d-b90c-92ee70db832d" (UID: "ead98060-0354-4a0d-b90c-92ee70db832d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:14.236580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.236539 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead98060-0354-4a0d-b90c-92ee70db832d-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:14.237912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.237885 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls" pod "ead98060-0354-4a0d-b90c-92ee70db832d" (UID: "ead98060-0354-4a0d-b90c-92ee70db832d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:14.238156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.238128 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-metrics-certs" pod "ead98060-0354-4a0d-b90c-92ee70db832d" (UID: "ead98060-0354-4a0d-b90c-92ee70db832d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:14.238867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.238841 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead98060-0354-4a0d-b90c-92ee70db832d-kube-api-access-vsd9r" pod "ead98060-0354-4a0d-b90c-92ee70db832d" (UID: "ead98060-0354-4a0d-b90c-92ee70db832d"). InnerVolumeSpecName "kube-api-access-vsd9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:14.337739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.337701 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:14.337739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.337736 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead98060-0354-4a0d-b90c-92ee70db832d-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:14.337968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:14.337749 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsd9r\" (UniqueName: \"kubernetes.io/projected/ead98060-0354-4a0d-b90c-92ee70db832d-kube-api-access-vsd9r\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:15.091463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:15.091410 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" Nov 01 16:33:15.091463 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:15.091444 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx" event={"ID":"ead98060-0354-4a0d-b90c-92ee70db832d","Type":"ContainerDied","Data":"1b3fddaae6b9de9172d1a00db9885cd79a905a066a920b450a133b10de36a170"} Nov 01 16:33:15.092028 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:15.091505 4911 scope.go:122] "RemoveContainer" containerID="2c5cb48a550ce4d9e7a72328476ebaf79af2134766a4b982bcbcb2b0d3627927" Nov 01 16:33:15.115734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:15.115703 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx"] Nov 01 16:33:15.119750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:15.119726 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a482b-746ff5cbfd-s85cx"] Nov 01 16:33:15.980664 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:15.980630 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead98060-0354-4a0d-b90c-92ee70db832d" path="/var/lib/kubelet/pods/ead98060-0354-4a0d-b90c-92ee70db832d/volumes" Nov 01 16:33:20.299540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.299471 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" containerID="cri-o://0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6" gracePeriod=45 Nov 01 16:33:20.645501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.645469 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:33:20.723364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.723247 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be0b5d67-06b1-4084-9754-3778e9a728d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/be0b5d67-06b1-4084-9754-3778e9a728d8-tls-certs\") pod \"be0b5d67-06b1-4084-9754-3778e9a728d8\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " Nov 01 16:33:20.723364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.723307 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-model-cache\") pod \"be0b5d67-06b1-4084-9754-3778e9a728d8\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " Nov 01 16:33:20.723703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.723373 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0b5d67-06b1-4084-9754-3778e9a728d8-kube-api-access-zwnmw\" (UniqueName: \"kubernetes.io/projected/be0b5d67-06b1-4084-9754-3778e9a728d8-kube-api-access-zwnmw\") pod \"be0b5d67-06b1-4084-9754-3778e9a728d8\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " Nov 01 16:33:20.723703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.723558 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-dshm\") pod \"be0b5d67-06b1-4084-9754-3778e9a728d8\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " Nov 01 16:33:20.723703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.723596 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-home\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-home\") pod \"be0b5d67-06b1-4084-9754-3778e9a728d8\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " Nov 01 16:33:20.723703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.723701 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-tmp-dir\") pod \"be0b5d67-06b1-4084-9754-3778e9a728d8\" (UID: \"be0b5d67-06b1-4084-9754-3778e9a728d8\") " Nov 01 16:33:20.724619 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.724588 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-tmp-dir" pod "be0b5d67-06b1-4084-9754-3778e9a728d8" (UID: "be0b5d67-06b1-4084-9754-3778e9a728d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.725592 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.725537 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-model-cache" pod "be0b5d67-06b1-4084-9754-3778e9a728d8" (UID: "be0b5d67-06b1-4084-9754-3778e9a728d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.725988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.725966 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-home" pod "be0b5d67-06b1-4084-9754-3778e9a728d8" (UID: "be0b5d67-06b1-4084-9754-3778e9a728d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.728828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.728795 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-dshm" pod "be0b5d67-06b1-4084-9754-3778e9a728d8" (UID: "be0b5d67-06b1-4084-9754-3778e9a728d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.729032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.729012 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0b5d67-06b1-4084-9754-3778e9a728d8-tls-certs" pod "be0b5d67-06b1-4084-9754-3778e9a728d8" (UID: "be0b5d67-06b1-4084-9754-3778e9a728d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:20.736175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.736145 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0b5d67-06b1-4084-9754-3778e9a728d8-kube-api-access-zwnmw" pod "be0b5d67-06b1-4084-9754-3778e9a728d8" (UID: "be0b5d67-06b1-4084-9754-3778e9a728d8"). InnerVolumeSpecName "kube-api-access-zwnmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:20.826140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.826032 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.826140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.826079 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.826140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.826090 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.826140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.826099 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0b5d67-06b1-4084-9754-3778e9a728d8-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.826140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.826108 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0b5d67-06b1-4084-9754-3778e9a728d8-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.826140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:20.826117 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnmw\" (UniqueName: \"kubernetes.io/projected/be0b5d67-06b1-4084-9754-3778e9a728d8-kube-api-access-zwnmw\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:21.132765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.132651 4911 generic.go:356] "Generic (PLEG): container finished" podID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerID="0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6" exitCode=0 Nov 01 16:33:21.132765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.132711 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" Nov 01 16:33:21.132765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.132738 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" event={"ID":"be0b5d67-06b1-4084-9754-3778e9a728d8","Type":"ContainerDied","Data":"0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6"} Nov 01 16:33:21.133030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.132779 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l" event={"ID":"be0b5d67-06b1-4084-9754-3778e9a728d8","Type":"ContainerDied","Data":"cf0a0ef9befc653ca1d8203daaf4b9d2279a4ec6d4cc53495af9b156c0816a25"} Nov 01 16:33:21.133030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.132797 4911 scope.go:122] "RemoveContainer" containerID="0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6" Nov 01 16:33:21.144836 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.144814 4911 scope.go:122] "RemoveContainer" containerID="0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6" Nov 01 16:33:21.145164 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:21.145140 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6\": container with ID starting with 0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6 not found: ID does not exist" containerID="0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6" Nov 01 16:33:21.145258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.145172 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6"} err="failed to get container status \"0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6\": rpc error: code = NotFound desc = could not find container \"0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6\": container with ID starting with 0e7c8f8ef76dca35b105dc55b9814d058ad0d99fe03317e6a9ee136a04e6c4a6 not found: ID does not exist" Nov 01 16:33:21.158501 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.158473 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l"] Nov 01 16:33:21.161436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.161410 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/distinct-2-w3-bd831a-kserve-7555456544-48p6l"] Nov 01 16:33:21.968506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:21.968461 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" path="/var/lib/kubelet/pods/be0b5d67-06b1-4084-9754-3778e9a728d8/volumes" Nov 01 16:33:22.413176 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:22.413116 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv"] Nov 01 16:33:22.413580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:22.413543 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" containerID="cri-o://71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587" gracePeriod=30 Nov 01 16:33:22.425932 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:22.425902 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j"] Nov 01 16:33:22.426143 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:22.426115 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" containerID="cri-o://1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6" gracePeriod=30 Nov 01 16:33:22.430824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:22.430796 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz"] Nov 01 16:33:22.431062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:22.431019 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" containerID="cri-o://be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b" gracePeriod=30 Nov 01 16:33:25.147992 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:25.147946 4911 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:33:25.148408 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:25.148027 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:33:27.753368 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.753331 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:33:27.806694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.806657 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:33:27.820868 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.820835 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:33:27.836808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.836770 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-kube-api-access-jpdkj\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-kube-api-access-jpdkj\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.836989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.836871 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-certs\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.836989 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.836929 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-socket\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.837137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.837025 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-token\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-token\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.837204 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.837193 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-podinfo\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.837260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.837250 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istiod-ca-cert\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.837329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.837311 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-credential-socket\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.837393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.837367 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-data\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.837440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.837403 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-envoy\") pod \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\" (UID: \"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2\") " Nov 01 16:33:27.838329 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.838294 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-certs" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.838474 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.838450 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istiod-ca-cert" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.838712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.838645 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-socket" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.838712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.838695 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-data" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.838831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.838727 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-credential-socket" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.840441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.840417 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-envoy" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.840700 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.840665 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-kube-api-access-jpdkj" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "kube-api-access-jpdkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.842069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.842022 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-token" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.842172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.842146 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-podinfo" pod "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" (UID: "dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.938760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.938719 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-kube-api-access-hsjjc\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-kube-api-access-hsjjc\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.938944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.938790 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-podinfo\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.938944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.938868 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-credential-socket\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.938944 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.938929 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istiod-ca-cert\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.938973 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-envoy\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939016 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392c10c3-f485-4701-b2f0-5de1966b35bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392c10c3-f485-4701-b2f0-5de1966b35bf-istiod-ca-cert\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.939205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939091 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-data\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939124 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-podinfo\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939183 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-socket\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.939383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939232 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-certs\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939290 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-certs\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.939383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939302 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-credential-socket" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.939383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939330 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-credential-socket\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939380 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-envoy\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.939668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939386 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istiod-ca-cert" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.939668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939421 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-socket\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939454 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-token\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-token\") pod \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\" (UID: \"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe\") " Nov 01 16:33:27.939668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939481 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-token\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-token\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.939668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939520 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-data\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.939668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939578 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-kube-api-access-l8jvq\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-kube-api-access-l8jvq\") pod \"392c10c3-f485-4701-b2f0-5de1966b35bf\" (UID: \"392c10c3-f485-4701-b2f0-5de1966b35bf\") " Nov 01 16:33:27.939668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939618 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-certs" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.940069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939870 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392c10c3-f485-4701-b2f0-5de1966b35bf-istiod-ca-cert" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.940069 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.939876 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-data" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.940187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940083 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-socket" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.940187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940104 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-socket" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940203 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940226 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940271 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940287 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940302 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdkj\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-kube-api-access-jpdkj\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940316 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-data" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940319 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392c10c3-f485-4701-b2f0-5de1966b35bf-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940359 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940380 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940397 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940413 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940429 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940468 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940484 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940498 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940502 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-credential-socket" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940511 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940526 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.940946 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-certs" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.941620 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-podinfo" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.942153 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-podinfo" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.942163 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-kube-api-access-hsjjc" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "kube-api-access-hsjjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.942355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.942273 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-token" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.943502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.942569 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-envoy" pod "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" (UID: "73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.943502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.942745 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-token" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.943502 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.943246 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-envoy" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.943758 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:27.943732 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-kube-api-access-l8jvq" pod "392c10c3-f485-4701-b2f0-5de1966b35bf" (UID: "392c10c3-f485-4701-b2f0-5de1966b35bf"). InnerVolumeSpecName "kube-api-access-l8jvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:28.041590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041551 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041586 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041598 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041606 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041616 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041625 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041640 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041651 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041660 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8jvq\" (UniqueName: \"kubernetes.io/projected/392c10c3-f485-4701-b2f0-5de1966b35bf-kube-api-access-l8jvq\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041669 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsjjc\" (UniqueName: \"kubernetes.io/projected/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe-kube-api-access-hsjjc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.041678 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392c10c3-f485-4701-b2f0-5de1966b35bf-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.187436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.187397 4911 generic.go:356] "Generic (PLEG): container finished" podID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerID="71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587" exitCode=0 Nov 01 16:33:28.187645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.187452 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" Nov 01 16:33:28.187645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.187489 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" event={"ID":"392c10c3-f485-4701-b2f0-5de1966b35bf","Type":"ContainerDied","Data":"71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587"} Nov 01 16:33:28.187645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.187571 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv" event={"ID":"392c10c3-f485-4701-b2f0-5de1966b35bf","Type":"ContainerDied","Data":"6df980a57657b58672b6a0ec0954a24abf5a2921ce667cdae1270bc65f8eb079"} Nov 01 16:33:28.187645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.187607 4911 scope.go:122] "RemoveContainer" containerID="71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587" Nov 01 16:33:28.189769 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.189745 4911 generic.go:356] "Generic (PLEG): container finished" podID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerID="1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6" exitCode=0 Nov 01 16:33:28.189915 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.189883 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" event={"ID":"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2","Type":"ContainerDied","Data":"1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6"} Nov 01 16:33:28.189988 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.189930 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" event={"ID":"dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2","Type":"ContainerDied","Data":"477f23a028168a46c626f156157abc3d5df780a3853f14031734128e045ad3f1"} Nov 01 16:33:28.190207 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.190187 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j" Nov 01 16:33:28.193351 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.193322 4911 generic.go:356] "Generic (PLEG): container finished" podID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerID="be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b" exitCode=0 Nov 01 16:33:28.193461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.193354 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" event={"ID":"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe","Type":"ContainerDied","Data":"be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b"} Nov 01 16:33:28.193461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.193394 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" event={"ID":"73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe","Type":"ContainerDied","Data":"ab6a8dbd2f5be63a3daef824eb04226133bdc927bf2dc619efdd018feb85d303"} Nov 01 16:33:28.193578 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.193464 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz" Nov 01 16:33:28.207403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.207378 4911 scope.go:122] "RemoveContainer" containerID="71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587" Nov 01 16:33:28.207781 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:28.207748 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587\": container with ID starting with 71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587 not found: ID does not exist" containerID="71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587" Nov 01 16:33:28.207892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.207793 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587"} err="failed to get container status \"71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587\": rpc error: code = NotFound desc = could not find container \"71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587\": container with ID starting with 71c089d0b264d69df39e684595ec3ee5f864637f27a7292fd1489e4f3e676587 not found: ID does not exist" Nov 01 16:33:28.207892 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.207817 4911 scope.go:122] "RemoveContainer" containerID="1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6" Nov 01 16:33:28.220754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.220730 4911 scope.go:122] "RemoveContainer" containerID="1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6" Nov 01 16:33:28.221540 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:28.221517 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6\": container with ID starting with 1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6 not found: ID does not exist" containerID="1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6" Nov 01 16:33:28.221650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.221549 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6"} err="failed to get container status \"1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6\": rpc error: code = NotFound desc = could not find container \"1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6\": container with ID starting with 1815af98ea8e299e4f5394d74d40e642171816c2a23519c7916239ce32937dd6 not found: ID does not exist" Nov 01 16:33:28.221650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.221571 4911 scope.go:122] "RemoveContainer" containerID="be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b" Nov 01 16:33:28.221650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.221602 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv"] Nov 01 16:33:28.223302 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.223281 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-9grjv"] Nov 01 16:33:28.232327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.232277 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j"] Nov 01 16:33:28.233083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.233069 4911 scope.go:122] "RemoveContainer" containerID="21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c" Nov 01 16:33:28.240739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.240713 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-x5c4j"] Nov 01 16:33:28.245653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.245630 4911 scope.go:122] "RemoveContainer" containerID="be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b" Nov 01 16:33:28.245994 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:28.245970 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b\": container with ID starting with be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b not found: ID does not exist" containerID="be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b" Nov 01 16:33:28.246096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.246006 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b"} err="failed to get container status \"be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b\": rpc error: code = NotFound desc = could not find container \"be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b\": container with ID starting with be74a3f785189c8a4f3aaccc9ef7fe29dd8faf057921748a79403aab08008e6b not found: ID does not exist" Nov 01 16:33:28.246096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.246030 4911 scope.go:122] "RemoveContainer" containerID="21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c" Nov 01 16:33:28.246356 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:28.246330 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c\": container with ID starting with 21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c not found: ID does not exist" containerID="21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c" Nov 01 16:33:28.246455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.246362 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c"} err="failed to get container status \"21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c\": rpc error: code = NotFound desc = could not find container \"21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c\": container with ID starting with 21d6693d5a52395a2a092ec89da6792a13d8bf6ad7f92ed0cd800d984d91b98c not found: ID does not exist" Nov 01 16:33:28.254540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.252321 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz"] Nov 01 16:33:28.257149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:28.257125 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-6g9zz"] Nov 01 16:33:29.971386 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:29.971349 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" path="/var/lib/kubelet/pods/392c10c3-f485-4701-b2f0-5de1966b35bf/volumes" Nov 01 16:33:29.972077 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:29.972028 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" path="/var/lib/kubelet/pods/73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe/volumes" Nov 01 16:33:29.972692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:29.972667 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" path="/var/lib/kubelet/pods/dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2/volumes" Nov 01 16:33:30.792057 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:30.791994 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv"] Nov 01 16:33:30.792341 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:30.792294 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" podUID="6f2c301e-c797-47bf-81e6-16251407a79e" containerName="maas-api" containerID="cri-o://f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549" gracePeriod=30 Nov 01 16:33:31.199472 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.199441 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 16:33:31.218833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.218799 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f2c301e-c797-47bf-81e6-16251407a79e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2c301e-c797-47bf-81e6-16251407a79e-openshift-service-ca\") pod \"6f2c301e-c797-47bf-81e6-16251407a79e\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " Nov 01 16:33:31.219034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.218859 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-metrics-certs\") pod \"6f2c301e-c797-47bf-81e6-16251407a79e\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " Nov 01 16:33:31.219034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.218893 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls\") pod \"6f2c301e-c797-47bf-81e6-16251407a79e\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " Nov 01 16:33:31.219208 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.219033 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2c301e-c797-47bf-81e6-16251407a79e-kube-api-access-8bh5b\" (UniqueName: \"kubernetes.io/projected/6f2c301e-c797-47bf-81e6-16251407a79e-kube-api-access-8bh5b\") pod \"6f2c301e-c797-47bf-81e6-16251407a79e\" (UID: \"6f2c301e-c797-47bf-81e6-16251407a79e\") " Nov 01 16:33:31.219303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.219265 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2c301e-c797-47bf-81e6-16251407a79e-openshift-service-ca" pod "6f2c301e-c797-47bf-81e6-16251407a79e" (UID: "6f2c301e-c797-47bf-81e6-16251407a79e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:31.219676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.219636 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2c301e-c797-47bf-81e6-16251407a79e-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:31.225071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.221784 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls" pod "6f2c301e-c797-47bf-81e6-16251407a79e" (UID: "6f2c301e-c797-47bf-81e6-16251407a79e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:31.225071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.221818 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-metrics-certs" pod "6f2c301e-c797-47bf-81e6-16251407a79e" (UID: "6f2c301e-c797-47bf-81e6-16251407a79e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:31.225071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.221911 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2c301e-c797-47bf-81e6-16251407a79e-kube-api-access-8bh5b" pod "6f2c301e-c797-47bf-81e6-16251407a79e" (UID: "6f2c301e-c797-47bf-81e6-16251407a79e"). InnerVolumeSpecName "kube-api-access-8bh5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:31.227516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.227490 4911 generic.go:356] "Generic (PLEG): container finished" podID="6f2c301e-c797-47bf-81e6-16251407a79e" containerID="f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549" exitCode=0 Nov 01 16:33:31.227652 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.227563 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" Nov 01 16:33:31.227719 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.227689 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" event={"ID":"6f2c301e-c797-47bf-81e6-16251407a79e","Type":"ContainerDied","Data":"f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549"} Nov 01 16:33:31.227786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.227744 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv" event={"ID":"6f2c301e-c797-47bf-81e6-16251407a79e","Type":"ContainerDied","Data":"ab6114c380068986b76a96eb751e958614e325e67f1db456365b86942a86d979"} Nov 01 16:33:31.227786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.227778 4911 scope.go:122] "RemoveContainer" containerID="f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549" Nov 01 16:33:31.254013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.253991 4911 scope.go:122] "RemoveContainer" containerID="f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549" Nov 01 16:33:31.254472 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:31.254448 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549\": container with ID starting with f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549 not found: ID does not exist" containerID="f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549" Nov 01 16:33:31.254565 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.254481 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549"} err="failed to get container status \"f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549\": rpc error: code = NotFound desc = could not find container \"f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549\": container with ID starting with f90bd4026b46be910ad2b7a2868501f1811f349c44e329c388be360143942549 not found: ID does not exist" Nov 01 16:33:31.263337 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.263305 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv"] Nov 01 16:33:31.266512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.266487 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bd831a-57b7989d7f-b6lpv"] Nov 01 16:33:31.320556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.320481 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bh5b\" (UniqueName: \"kubernetes.io/projected/6f2c301e-c797-47bf-81e6-16251407a79e-kube-api-access-8bh5b\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:31.320556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.320509 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:31.320556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.320520 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2c301e-c797-47bf-81e6-16251407a79e-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:31.966613 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:31.966577 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2c301e-c797-47bf-81e6-16251407a79e" path="/var/lib/kubelet/pods/6f2c301e-c797-47bf-81e6-16251407a79e/volumes" Nov 01 16:33:38.312062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:38.311818 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh"] Nov 01 16:33:38.312639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:38.312505 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" containerID="cri-o://f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987" gracePeriod=30 Nov 01 16:33:38.977507 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:38.977468 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs"] Nov 01 16:33:38.977750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:38.977720 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" podUID="e3b19626-9d05-4f9c-835a-e1103e47e657" containerName="maas-api" containerID="cri-o://aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337" gracePeriod=30 Nov 01 16:33:39.367432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.367390 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:33:39.370394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.370362 4911 generic.go:356] "Generic (PLEG): container finished" podID="e3b19626-9d05-4f9c-835a-e1103e47e657" containerID="aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337" exitCode=0 Nov 01 16:33:39.370621 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.370457 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" event={"ID":"e3b19626-9d05-4f9c-835a-e1103e47e657","Type":"ContainerDied","Data":"aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337"} Nov 01 16:33:39.373484 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.373446 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" event={"ID":"e3b19626-9d05-4f9c-835a-e1103e47e657","Type":"ContainerDied","Data":"5a78f548448ad928a6191e940152e5fc31daad207732d582c1bcbfd8b7099a6e"} Nov 01 16:33:39.374268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.373652 4911 scope.go:122] "RemoveContainer" containerID="aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337" Nov 01 16:33:39.386407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.386384 4911 scope.go:122] "RemoveContainer" containerID="aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337" Nov 01 16:33:39.386689 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:39.386664 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337\": container with ID starting with aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337 not found: ID does not exist" containerID="aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337" Nov 01 16:33:39.386747 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.386700 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337"} err="failed to get container status \"aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337\": rpc error: code = NotFound desc = could not find container \"aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337\": container with ID starting with aefb8a408bce9183e4110e5566b05e6f61ae55ce2f11990e3d14608bc36fb337 not found: ID does not exist" Nov 01 16:33:39.446381 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.446342 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-metrics-certs\") pod \"e3b19626-9d05-4f9c-835a-e1103e47e657\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " Nov 01 16:33:39.446598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.446500 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-maas-api-tls\") pod \"e3b19626-9d05-4f9c-835a-e1103e47e657\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " Nov 01 16:33:39.446598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.446558 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b19626-9d05-4f9c-835a-e1103e47e657-kube-api-access-lstwt\" (UniqueName: \"kubernetes.io/projected/e3b19626-9d05-4f9c-835a-e1103e47e657-kube-api-access-lstwt\") pod \"e3b19626-9d05-4f9c-835a-e1103e47e657\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " Nov 01 16:33:39.446598 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.446593 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b19626-9d05-4f9c-835a-e1103e47e657-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b19626-9d05-4f9c-835a-e1103e47e657-openshift-service-ca\") pod \"e3b19626-9d05-4f9c-835a-e1103e47e657\" (UID: \"e3b19626-9d05-4f9c-835a-e1103e47e657\") " Nov 01 16:33:39.447373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.447338 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b19626-9d05-4f9c-835a-e1103e47e657-openshift-service-ca" pod "e3b19626-9d05-4f9c-835a-e1103e47e657" (UID: "e3b19626-9d05-4f9c-835a-e1103e47e657"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:39.449779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.449755 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-maas-api-tls" pod "e3b19626-9d05-4f9c-835a-e1103e47e657" (UID: "e3b19626-9d05-4f9c-835a-e1103e47e657"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:39.449894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.449846 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-metrics-certs" pod "e3b19626-9d05-4f9c-835a-e1103e47e657" (UID: "e3b19626-9d05-4f9c-835a-e1103e47e657"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:39.451580 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.451543 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b19626-9d05-4f9c-835a-e1103e47e657-kube-api-access-lstwt" pod "e3b19626-9d05-4f9c-835a-e1103e47e657" (UID: "e3b19626-9d05-4f9c-835a-e1103e47e657"). InnerVolumeSpecName "kube-api-access-lstwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:39.547726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.547638 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b19626-9d05-4f9c-835a-e1103e47e657-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:39.547726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.547676 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:39.547726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.547687 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3b19626-9d05-4f9c-835a-e1103e47e657-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:39.547726 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:39.547696 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lstwt\" (UniqueName: \"kubernetes.io/projected/e3b19626-9d05-4f9c-835a-e1103e47e657-kube-api-access-lstwt\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:40.377703 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:40.377671 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs" Nov 01 16:33:40.398414 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:40.398364 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs"] Nov 01 16:33:40.411342 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:40.411308 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f01de5-6877c9f7b4-btsfs"] Nov 01 16:33:40.867093 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:40.867020 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:33:41.969904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:41.969858 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b19626-9d05-4f9c-835a-e1103e47e657" path="/var/lib/kubelet/pods/e3b19626-9d05-4f9c-835a-e1103e47e657/volumes" Nov 01 16:33:43.601789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.601727 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 16:33:43.694299 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694254 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72b79f2c-6850-468e-8994-0c409eead207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b79f2c-6850-468e-8994-0c409eead207-istio-podinfo\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.694455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694318 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-data\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.694455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694404 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-envoy\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.694590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694454 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-certs\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.694590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694559 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-credential-socket\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.694760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694594 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-istio-token\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-istio-token\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.694760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694704 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-kube-api-access-zmwgf\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-kube-api-access-zmwgf\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.694760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694708 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-data" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.694760 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694754 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-socket\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.695172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694800 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b79f2c-6850-468e-8994-0c409eead207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b79f2c-6850-468e-8994-0c409eead207-istiod-ca-cert\") pod \"72b79f2c-6850-468e-8994-0c409eead207\" (UID: \"72b79f2c-6850-468e-8994-0c409eead207\") " Nov 01 16:33:43.695172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.694814 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-credential-socket" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.695172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.695054 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-socket" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.695172 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.695115 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-certs" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.695451 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.695429 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.695521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.695450 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b79f2c-6850-468e-8994-0c409eead207-istiod-ca-cert" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:43.695521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.695500 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.695638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.695520 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.695638 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.695535 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.697469 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.697441 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72b79f2c-6850-468e-8994-0c409eead207-istio-podinfo" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:43.697635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.697614 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-kube-api-access-zmwgf" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "kube-api-access-zmwgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:43.698489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.698464 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-envoy" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.698919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.698886 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-istio-token" pod "72b79f2c-6850-468e-8994-0c409eead207" (UID: "72b79f2c-6850-468e-8994-0c409eead207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:43.796962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.796917 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b79f2c-6850-468e-8994-0c409eead207-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.796962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.796966 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b79f2c-6850-468e-8994-0c409eead207-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.797228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.796982 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.797228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.796996 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwgf\" (UniqueName: \"kubernetes.io/projected/72b79f2c-6850-468e-8994-0c409eead207-kube-api-access-zmwgf\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.797228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:43.797010 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b79f2c-6850-468e-8994-0c409eead207-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:44.404354 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.404321 4911 generic.go:356] "Generic (PLEG): container finished" podID="72b79f2c-6850-468e-8994-0c409eead207" containerID="f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987" exitCode=0 Nov 01 16:33:44.404567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.404386 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" Nov 01 16:33:44.404567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.404407 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" event={"ID":"72b79f2c-6850-468e-8994-0c409eead207","Type":"ContainerDied","Data":"f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987"} Nov 01 16:33:44.404567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.404470 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh" event={"ID":"72b79f2c-6850-468e-8994-0c409eead207","Type":"ContainerDied","Data":"dc442a1e29c51d4c3655602d471dcadacc164bd9ef18afa3ea2a9cd9129c6213"} Nov 01 16:33:44.404567 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.404502 4911 scope.go:122] "RemoveContainer" containerID="f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987" Nov 01 16:33:44.415605 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.415582 4911 scope.go:122] "RemoveContainer" containerID="ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9" Nov 01 16:33:44.424805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.424780 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh"] Nov 01 16:33:44.425887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.425860 4911 scope.go:122] "RemoveContainer" containerID="f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987" Nov 01 16:33:44.426181 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:44.426158 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987\": container with ID starting with f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987 not found: ID does not exist" containerID="f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987" Nov 01 16:33:44.426275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.426192 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987"} err="failed to get container status \"f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987\": rpc error: code = NotFound desc = could not find container \"f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987\": container with ID starting with f213a599075ddff1e909eb179ac133f93779907dfb8e6da569ce047854535987 not found: ID does not exist" Nov 01 16:33:44.426275 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.426216 4911 scope.go:122] "RemoveContainer" containerID="ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9" Nov 01 16:33:44.426505 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:33:44.426489 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9\": container with ID starting with ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9 not found: ID does not exist" containerID="ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9" Nov 01 16:33:44.426547 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.426515 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9"} err="failed to get container status \"ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9\": rpc error: code = NotFound desc = could not find container \"ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9\": container with ID starting with ef6c6aff205bbd4b8cd2b0ede492fbe060a60c19e641686d37aed39c37662ce9 not found: ID does not exist" Nov 01 16:33:44.431393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:44.431370 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-bnmxh"] Nov 01 16:33:45.974707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:45.974666 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b79f2c-6850-468e-8994-0c409eead207" path="/var/lib/kubelet/pods/72b79f2c-6850-468e-8994-0c409eead207/volumes" Nov 01 16:33:46.220083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:46.220031 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7"] Nov 01 16:33:46.220340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:46.220297 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" containerID="cri-o://67cda1fec47b3d2969ccb95f7a18357a2efc99e8a5023e47dc3a1dcf00e09a6e" gracePeriod=30 Nov 01 16:33:47.653215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:47.653175 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68"] Nov 01 16:33:47.667609 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:47.667572 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc"] Nov 01 16:33:47.677455 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:47.677348 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h"] Nov 01 16:33:51.454310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.454266 4911 generic.go:356] "Generic (PLEG): container finished" podID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerID="67cda1fec47b3d2969ccb95f7a18357a2efc99e8a5023e47dc3a1dcf00e09a6e" exitCode=0 Nov 01 16:33:51.454813 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.454335 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" event={"ID":"ccf9c983-3687-4898-b4bd-744157f4bdd2","Type":"ContainerDied","Data":"67cda1fec47b3d2969ccb95f7a18357a2efc99e8a5023e47dc3a1dcf00e09a6e"} Nov 01 16:33:51.527080 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.526988 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:33:51.623599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623561 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-socket\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.623784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623631 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-envoy\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.623784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623682 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-kube-api-access-5fh7v\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-kube-api-access-5fh7v\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.623784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623710 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5"] Nov 01 16:33:51.623784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623725 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-podinfo\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.623784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623765 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-credential-socket\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.624140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623836 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-certs\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.624140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623860 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-socket" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.624140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.623929 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-data\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.624140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624024 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-token\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-token\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.624140 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624098 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf9c983-3687-4898-b4bd-744157f4bdd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf9c983-3687-4898-b4bd-744157f4bdd2-istiod-ca-cert\") pod \"ccf9c983-3687-4898-b4bd-744157f4bdd2\" (UID: \"ccf9c983-3687-4898-b4bd-744157f4bdd2\") " Nov 01 16:33:51.624438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624144 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-certs" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.624438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624161 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-credential-socket" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.624438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624338 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-data" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.624611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624567 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.624611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624589 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.624611 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624603 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.624786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624616 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.624786 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.624655 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf9c983-3687-4898-b4bd-744157f4bdd2-istiod-ca-cert" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:51.626675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.626592 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-podinfo" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:51.626675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.626602 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-kube-api-access-5fh7v" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "kube-api-access-5fh7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:51.626882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.626735 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-token" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:51.626882 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.626814 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-envoy" pod "ccf9c983-3687-4898-b4bd-744157f4bdd2" (UID: "ccf9c983-3687-4898-b4bd-744157f4bdd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.725622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.725568 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.725622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.725601 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf9c983-3687-4898-b4bd-744157f4bdd2-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.725622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.725618 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.725622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.725632 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fh7v\" (UniqueName: \"kubernetes.io/projected/ccf9c983-3687-4898-b4bd-744157f4bdd2-kube-api-access-5fh7v\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.726453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:51.725642 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf9c983-3687-4898-b4bd-744157f4bdd2-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:52.462126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:52.462089 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" Nov 01 16:33:52.462126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:52.462114 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7" event={"ID":"ccf9c983-3687-4898-b4bd-744157f4bdd2","Type":"ContainerDied","Data":"63af40dabc3414e88cefcfecf7ccbafba974752befeb719c719be51b134441bc"} Nov 01 16:33:52.462731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:52.462179 4911 scope.go:122] "RemoveContainer" containerID="67cda1fec47b3d2969ccb95f7a18357a2efc99e8a5023e47dc3a1dcf00e09a6e" Nov 01 16:33:52.483312 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:52.483282 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7"] Nov 01 16:33:52.485948 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:52.485923 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-bqnc7"] Nov 01 16:33:53.967714 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:53.967676 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" path="/var/lib/kubelet/pods/ccf9c983-3687-4898-b4bd-744157f4bdd2/volumes" Nov 01 16:33:56.902670 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:56.902629 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx"] Nov 01 16:33:59.623809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.623769 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r"] Nov 01 16:33:59.624586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624555 4911 cpu_manager.go:414] "containerMap: removing container" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" Nov 01 16:33:59.624586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624586 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624601 4911 cpu_manager.go:414] "containerMap: removing container" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624611 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624635 4911 cpu_manager.go:414] "containerMap: removing container" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624645 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624662 4911 cpu_manager.go:414] "containerMap: removing container" podUID="6f2c301e-c797-47bf-81e6-16251407a79e" containerName="maas-api" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624670 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2c301e-c797-47bf-81e6-16251407a79e" containerName="maas-api" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624693 4911 cpu_manager.go:414] "containerMap: removing container" podUID="a8a96349-9c37-491b-b324-6bd81ed6f07d" containerName="main" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624701 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a96349-9c37-491b-b324-6bd81ed6f07d" containerName="main" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624717 4911 cpu_manager.go:414] "containerMap: removing container" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624724 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624735 4911 cpu_manager.go:414] "containerMap: removing container" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624742 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624758 4911 cpu_manager.go:414] "containerMap: removing container" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624765 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624778 4911 cpu_manager.go:414] "containerMap: removing container" podUID="c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" containerName="main" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624785 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" containerName="main" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624797 4911 cpu_manager.go:414] "containerMap: removing container" podUID="de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" containerName="maas-api" Nov 01 16:33:59.624804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624804 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624821 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624828 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624840 4911 cpu_manager.go:414] "containerMap: removing container" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624845 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624865 4911 cpu_manager.go:414] "containerMap: removing container" podUID="ead98060-0354-4a0d-b90c-92ee70db832d" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624871 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead98060-0354-4a0d-b90c-92ee70db832d" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624887 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e3b19626-9d05-4f9c-835a-e1103e47e657" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624892 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b19626-9d05-4f9c-835a-e1103e47e657" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624902 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.624908 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625003 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625012 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625019 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b79f2c-6850-468e-8994-0c409eead207" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625025 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2c301e-c797-47bf-81e6-16251407a79e" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625031 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a96349-9c37-491b-b324-6bd81ed6f07d" containerName="main" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625054 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="392c10c3-f485-4701-b2f0-5de1966b35bf" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625063 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ea6369-c5f4-4167-ae88-ff8e3c6bd0fe" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625071 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6e8c41-3ac7-40c1-bd1e-bd9c1b47cbd2" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625078 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50ef3d7-ab6e-4009-8a8d-78d80de3ead5" containerName="main" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625084 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0c13ae-cdf1-4eb3-a843-c07f0c3a7a47" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625090 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf9c983-3687-4898-b4bd-744157f4bdd2" containerName="istio-proxy" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625096 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0b5d67-06b1-4084-9754-3778e9a728d8" containerName="main" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625103 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead98060-0354-4a0d-b90c-92ee70db832d" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625110 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b19626-9d05-4f9c-835a-e1103e47e657" containerName="maas-api" Nov 01 16:33:59.625488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.625116 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ecd3e2-ea3f-461e-b436-36910fc14f95" containerName="istio-proxy" Nov 01 16:33:59.630448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.630420 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.637304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.637271 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r"] Nov 01 16:33:59.637653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.637633 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:33:59.641261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.641235 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:33:59.725370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.725328 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkn5p\" (UniqueName: \"kubernetes.io/projected/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-kube-api-access-mkn5p\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.725529 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.725503 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.725590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.725549 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.826858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.826820 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkn5p\" (UniqueName: \"kubernetes.io/projected/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-kube-api-access-mkn5p\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.827107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.826987 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.827107 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.827031 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.827841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.827820 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.829556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.829526 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.835694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.835675 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkn5p\" (UniqueName: \"kubernetes.io/projected/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-kube-api-access-mkn5p\") pod \"maas-api-revoke-sub-77d75604-9nq8r\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:33:59.955543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:33:59.955441 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:34:00.233701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:00.233665 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r"] Nov 01 16:34:00.235020 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:34:00.234987 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f9ee1e_83a0_4b6b_acb8_84d6ecf0b299.slice/crio-06f5a322f094f4b4cca97365c743a805f1263a6b7f3008acec439a2acc0f403a WatchSource:0}: Error finding container 06f5a322f094f4b4cca97365c743a805f1263a6b7f3008acec439a2acc0f403a: Status 404 returned error can't find the container with id 06f5a322f094f4b4cca97365c743a805f1263a6b7f3008acec439a2acc0f403a Nov 01 16:34:00.514694 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:00.514653 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" event={"ID":"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299","Type":"ContainerStarted","Data":"06f5a322f094f4b4cca97365c743a805f1263a6b7f3008acec439a2acc0f403a"} Nov 01 16:34:01.524083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:01.524010 4911 generic.go:356] "Generic (PLEG): container finished" podID="c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" containerID="d0f0f80afa6172297259155031ede4b9ad2ded64cdba2e3d5f9af802996a99c4" exitCode=0 Nov 01 16:34:01.524574 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:01.524106 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" event={"ID":"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299","Type":"ContainerDied","Data":"d0f0f80afa6172297259155031ede4b9ad2ded64cdba2e3d5f9af802996a99c4"} Nov 01 16:34:01.903875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:01.903770 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" podUID="e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" containerName="payload-processing" containerID="cri-o://e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7" gracePeriod=25 Nov 01 16:34:02.178456 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.178425 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 16:34:02.360415 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.360379 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-plugins-config-volume\") pod \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " Nov 01 16:34:02.360614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.360470 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-kube-api-access-z4jlz\" (UniqueName: \"kubernetes.io/projected/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-kube-api-access-z4jlz\") pod \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\" (UID: \"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e\") " Nov 01 16:34:02.360811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.360778 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-plugins-config-volume" pod "e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" (UID: "e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:02.362985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.362953 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-kube-api-access-z4jlz" pod "e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" (UID: "e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e"). InnerVolumeSpecName "kube-api-access-z4jlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:02.462345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.462260 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.462345 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.462291 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4jlz\" (UniqueName: \"kubernetes.io/projected/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e-kube-api-access-z4jlz\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.531447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.531412 4911 generic.go:356] "Generic (PLEG): container finished" podID="e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" containerID="e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7" exitCode=0 Nov 01 16:34:02.531946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.531454 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" Nov 01 16:34:02.531946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.531504 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" event={"ID":"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e","Type":"ContainerDied","Data":"e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7"} Nov 01 16:34:02.531946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.531576 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx" event={"ID":"e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e","Type":"ContainerDied","Data":"3e618b3977aac6dd2cf4f12c138d13d47bc56b6748eac92db5dcd47be5b55393"} Nov 01 16:34:02.531946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.531612 4911 scope.go:122] "RemoveContainer" containerID="e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7" Nov 01 16:34:02.543722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.543636 4911 scope.go:122] "RemoveContainer" containerID="e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7" Nov 01 16:34:02.547392 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:02.547357 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7\": container with ID starting with e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7 not found: ID does not exist" containerID="e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7" Nov 01 16:34:02.547539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.547401 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7"} err="failed to get container status \"e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7\": rpc error: code = NotFound desc = could not find container \"e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7\": container with ID starting with e68119f32f51574accbacae4b5a73bf1b3bb8349cdac0e1d7845da25a12744f7 not found: ID does not exist" Nov 01 16:34:02.560309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.560271 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx"] Nov 01 16:34:02.563654 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.563625 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9f4dc9-76d5dd8489-6bczx"] Nov 01 16:34:02.673343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.673313 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:34:02.675620 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.675591 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" containerID="cri-o://5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b" gracePeriod=45 Nov 01 16:34:02.693609 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.693574 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" containerID="cri-o://cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529" gracePeriod=45 Nov 01 16:34:02.699186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.699150 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" containerID="cri-o://cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c" gracePeriod=45 Nov 01 16:34:02.764975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.764938 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-openshift-ca\") pod \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " Nov 01 16:34:02.764975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.764982 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-xks-ca\") pod \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " Nov 01 16:34:02.765265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.765188 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-kube-api-access-mkn5p\" (UniqueName: \"kubernetes.io/projected/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-kube-api-access-mkn5p\") pod \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\" (UID: \"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299\") " Nov 01 16:34:02.765442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.765411 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-openshift-ca" pod "c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" (UID: "c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:02.765770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.765715 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.767740 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.767715 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-kube-api-access-mkn5p" pod "c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" (UID: "c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299"). InnerVolumeSpecName "kube-api-access-mkn5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:02.767843 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.767714 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-xks-ca" pod "c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" (UID: "c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:02.817826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.817786 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused" Nov 01 16:34:02.867277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.867238 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.867277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.867270 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkn5p\" (UniqueName: \"kubernetes.io/projected/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299-kube-api-access-mkn5p\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.996049 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:02.995663 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:34:03.032554 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.032522 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:34:03.081575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.081546 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:34:03.171258 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171225 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170897de-1a22-4f8f-be3b-d4b1b4322663-kube-api-access-wbgqf\" (UniqueName: \"kubernetes.io/projected/170897de-1a22-4f8f-be3b-d4b1b4322663-kube-api-access-wbgqf\") pod \"170897de-1a22-4f8f-be3b-d4b1b4322663\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " Nov 01 16:34:03.171452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171303 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-model-cache\") pod \"99a13fd4-ab3e-42a5-872e-8eccecef961c\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " Nov 01 16:34:03.171452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171411 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/170897de-1a22-4f8f-be3b-d4b1b4322663-tls-certs\" (UniqueName: \"kubernetes.io/secret/170897de-1a22-4f8f-be3b-d4b1b4322663-tls-certs\") pod \"170897de-1a22-4f8f-be3b-d4b1b4322663\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " Nov 01 16:34:03.171579 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171463 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-home\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-home\") pod \"99a13fd4-ab3e-42a5-872e-8eccecef961c\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " Nov 01 16:34:03.171579 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171518 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-model-cache" pod "99a13fd4-ab3e-42a5-872e-8eccecef961c" (UID: "99a13fd4-ab3e-42a5-872e-8eccecef961c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.171579 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171524 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-dshm\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-dshm\") pod \"170897de-1a22-4f8f-be3b-d4b1b4322663\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " Nov 01 16:34:03.171746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171621 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a13fd4-ab3e-42a5-872e-8eccecef961c-kube-api-access-xcd9c\" (UniqueName: \"kubernetes.io/projected/99a13fd4-ab3e-42a5-872e-8eccecef961c-kube-api-access-xcd9c\") pod \"99a13fd4-ab3e-42a5-872e-8eccecef961c\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " Nov 01 16:34:03.171746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171658 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a13fd4-ab3e-42a5-872e-8eccecef961c-tls-certs\" (UniqueName: \"kubernetes.io/secret/99a13fd4-ab3e-42a5-872e-8eccecef961c-tls-certs\") pod \"99a13fd4-ab3e-42a5-872e-8eccecef961c\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " Nov 01 16:34:03.171746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171695 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-dshm\") pod \"99a13fd4-ab3e-42a5-872e-8eccecef961c\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " Nov 01 16:34:03.171746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171717 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-home" pod "99a13fd4-ab3e-42a5-872e-8eccecef961c" (UID: "99a13fd4-ab3e-42a5-872e-8eccecef961c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.171746 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171734 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-model-cache\") pod \"170897de-1a22-4f8f-be3b-d4b1b4322663\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " Nov 01 16:34:03.172001 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171795 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-tmp-dir\") pod \"99a13fd4-ab3e-42a5-872e-8eccecef961c\" (UID: \"99a13fd4-ab3e-42a5-872e-8eccecef961c\") " Nov 01 16:34:03.172001 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171854 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-tmp-dir\") pod \"170897de-1a22-4f8f-be3b-d4b1b4322663\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " Nov 01 16:34:03.172001 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.171882 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-home\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-home\") pod \"170897de-1a22-4f8f-be3b-d4b1b4322663\" (UID: \"170897de-1a22-4f8f-be3b-d4b1b4322663\") " Nov 01 16:34:03.172323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.172303 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.172395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.172329 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.172395 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.172291 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-tmp-dir" pod "99a13fd4-ab3e-42a5-872e-8eccecef961c" (UID: "99a13fd4-ab3e-42a5-872e-8eccecef961c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.172543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.172500 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-model-cache" pod "170897de-1a22-4f8f-be3b-d4b1b4322663" (UID: "170897de-1a22-4f8f-be3b-d4b1b4322663"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.172612 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.172558 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-tmp-dir" pod "170897de-1a22-4f8f-be3b-d4b1b4322663" (UID: "170897de-1a22-4f8f-be3b-d4b1b4322663"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.172736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.172716 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-home" pod "170897de-1a22-4f8f-be3b-d4b1b4322663" (UID: "170897de-1a22-4f8f-be3b-d4b1b4322663"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.173955 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.173931 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-dshm" pod "170897de-1a22-4f8f-be3b-d4b1b4322663" (UID: "170897de-1a22-4f8f-be3b-d4b1b4322663"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.174195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.174162 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170897de-1a22-4f8f-be3b-d4b1b4322663-kube-api-access-wbgqf" pod "170897de-1a22-4f8f-be3b-d4b1b4322663" (UID: "170897de-1a22-4f8f-be3b-d4b1b4322663"). InnerVolumeSpecName "kube-api-access-wbgqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:03.174195 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.174185 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a13fd4-ab3e-42a5-872e-8eccecef961c-tls-certs" pod "99a13fd4-ab3e-42a5-872e-8eccecef961c" (UID: "99a13fd4-ab3e-42a5-872e-8eccecef961c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:03.174335 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.174205 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a13fd4-ab3e-42a5-872e-8eccecef961c-kube-api-access-xcd9c" pod "99a13fd4-ab3e-42a5-872e-8eccecef961c" (UID: "99a13fd4-ab3e-42a5-872e-8eccecef961c"). InnerVolumeSpecName "kube-api-access-xcd9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:03.174335 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.174178 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170897de-1a22-4f8f-be3b-d4b1b4322663-tls-certs" pod "170897de-1a22-4f8f-be3b-d4b1b4322663" (UID: "170897de-1a22-4f8f-be3b-d4b1b4322663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:03.174676 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.174661 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-dshm" pod "99a13fd4-ab3e-42a5-872e-8eccecef961c" (UID: "99a13fd4-ab3e-42a5-872e-8eccecef961c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.273552 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273463 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-tmp-dir\") pod \"9153161a-1192-4086-b3c5-8dcced55123f\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " Nov 01 16:34:03.273552 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273530 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9153161a-1192-4086-b3c5-8dcced55123f-kube-api-access-nmzds\" (UniqueName: \"kubernetes.io/projected/9153161a-1192-4086-b3c5-8dcced55123f-kube-api-access-nmzds\") pod \"9153161a-1192-4086-b3c5-8dcced55123f\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " Nov 01 16:34:03.273552 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273555 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-dshm\") pod \"9153161a-1192-4086-b3c5-8dcced55123f\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " Nov 01 16:34:03.273852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273574 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-model-cache\") pod \"9153161a-1192-4086-b3c5-8dcced55123f\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " Nov 01 16:34:03.273852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273616 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-home\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-home\") pod \"9153161a-1192-4086-b3c5-8dcced55123f\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " Nov 01 16:34:03.273852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273645 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9153161a-1192-4086-b3c5-8dcced55123f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9153161a-1192-4086-b3c5-8dcced55123f-tls-certs\") pod \"9153161a-1192-4086-b3c5-8dcced55123f\" (UID: \"9153161a-1192-4086-b3c5-8dcced55123f\") " Nov 01 16:34:03.273852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273812 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-tmp-dir" pod "9153161a-1192-4086-b3c5-8dcced55123f" (UID: "9153161a-1192-4086-b3c5-8dcced55123f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.274094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273873 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-home" pod "9153161a-1192-4086-b3c5-8dcced55123f" (UID: "9153161a-1192-4086-b3c5-8dcced55123f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.274094 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.273891 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-model-cache" pod "9153161a-1192-4086-b3c5-8dcced55123f" (UID: "9153161a-1192-4086-b3c5-8dcced55123f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.274211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274187 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274216 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274233 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274246 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274268 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274261 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbgqf\" (UniqueName: \"kubernetes.io/projected/170897de-1a22-4f8f-be3b-d4b1b4322663-kube-api-access-wbgqf\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274277 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274290 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170897de-1a22-4f8f-be3b-d4b1b4322663-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274302 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274317 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcd9c\" (UniqueName: \"kubernetes.io/projected/99a13fd4-ab3e-42a5-872e-8eccecef961c-kube-api-access-xcd9c\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274330 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a13fd4-ab3e-42a5-872e-8eccecef961c-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274342 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a13fd4-ab3e-42a5-872e-8eccecef961c-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274356 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/170897de-1a22-4f8f-be3b-d4b1b4322663-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.274403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.274371 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.275717 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.275692 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-dshm" pod "9153161a-1192-4086-b3c5-8dcced55123f" (UID: "9153161a-1192-4086-b3c5-8dcced55123f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.275804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.275716 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9153161a-1192-4086-b3c5-8dcced55123f-tls-certs" pod "9153161a-1192-4086-b3c5-8dcced55123f" (UID: "9153161a-1192-4086-b3c5-8dcced55123f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:03.275804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.275720 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9153161a-1192-4086-b3c5-8dcced55123f-kube-api-access-nmzds" pod "9153161a-1192-4086-b3c5-8dcced55123f" (UID: "9153161a-1192-4086-b3c5-8dcced55123f"). InnerVolumeSpecName "kube-api-access-nmzds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:03.374947 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.374911 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzds\" (UniqueName: \"kubernetes.io/projected/9153161a-1192-4086-b3c5-8dcced55123f-kube-api-access-nmzds\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.374947 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.374943 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9153161a-1192-4086-b3c5-8dcced55123f-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.374947 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.374953 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9153161a-1192-4086-b3c5-8dcced55123f-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.539880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.539781 4911 generic.go:356] "Generic (PLEG): container finished" podID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerID="cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529" exitCode=0 Nov 01 16:34:03.539880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.539826 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" Nov 01 16:34:03.539880 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.539866 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" event={"ID":"170897de-1a22-4f8f-be3b-d4b1b4322663","Type":"ContainerDied","Data":"cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529"} Nov 01 16:34:03.540448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.539914 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc" event={"ID":"170897de-1a22-4f8f-be3b-d4b1b4322663","Type":"ContainerDied","Data":"1638b85f6ea7789f9a47fe66d50ff4f4cd9ebe38c74a1e1db7bf3d3930c81497"} Nov 01 16:34:03.540448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.539939 4911 scope.go:122] "RemoveContainer" containerID="cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529" Nov 01 16:34:03.541403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.541379 4911 generic.go:356] "Generic (PLEG): container finished" podID="9153161a-1192-4086-b3c5-8dcced55123f" containerID="cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c" exitCode=0 Nov 01 16:34:03.541517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.541440 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" Nov 01 16:34:03.541517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.541456 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" event={"ID":"9153161a-1192-4086-b3c5-8dcced55123f","Type":"ContainerDied","Data":"cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c"} Nov 01 16:34:03.541517 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.541485 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h" event={"ID":"9153161a-1192-4086-b3c5-8dcced55123f","Type":"ContainerDied","Data":"e23d33b19647e35f984c00c202baa24a0de88a9ab8f3b20e1812e4eceedc2541"} Nov 01 16:34:03.545059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.542842 4911 generic.go:356] "Generic (PLEG): container finished" podID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerID="5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b" exitCode=0 Nov 01 16:34:03.545059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.542872 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" event={"ID":"99a13fd4-ab3e-42a5-872e-8eccecef961c","Type":"ContainerDied","Data":"5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b"} Nov 01 16:34:03.545059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.542893 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" Nov 01 16:34:03.545059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.542900 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68" event={"ID":"99a13fd4-ab3e-42a5-872e-8eccecef961c","Type":"ContainerDied","Data":"cbf8f41b5504acb6171996b5f3656728b788e3bce91efdb58fc7d64b011b941a"} Nov 01 16:34:03.545059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.544940 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" Nov 01 16:34:03.545059 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.544986 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r" event={"ID":"c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299","Type":"ContainerDied","Data":"06f5a322f094f4b4cca97365c743a805f1263a6b7f3008acec439a2acc0f403a"} Nov 01 16:34:03.545432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.545034 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f5a322f094f4b4cca97365c743a805f1263a6b7f3008acec439a2acc0f403a" Nov 01 16:34:03.553684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.553417 4911 scope.go:122] "RemoveContainer" containerID="cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529" Nov 01 16:34:03.554138 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:03.554108 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529\": container with ID starting with cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529 not found: ID does not exist" containerID="cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529" Nov 01 16:34:03.554310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.554146 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529"} err="failed to get container status \"cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529\": rpc error: code = NotFound desc = could not find container \"cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529\": container with ID starting with cdf77a01b6501f541603a85eca6f52589872722661ac7917b753d472e642d529 not found: ID does not exist" Nov 01 16:34:03.554310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.554171 4911 scope.go:122] "RemoveContainer" containerID="cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c" Nov 01 16:34:03.578382 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.578350 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68"] Nov 01 16:34:03.580187 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.580169 4911 scope.go:122] "RemoveContainer" containerID="cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c" Nov 01 16:34:03.580526 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:03.580503 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c\": container with ID starting with cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c not found: ID does not exist" containerID="cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c" Nov 01 16:34:03.580737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.580535 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c"} err="failed to get container status \"cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c\": rpc error: code = NotFound desc = could not find container \"cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c\": container with ID starting with cb0e6a9c461f8ae6d9a25e3774da4f974cf6f87eee9e4a29023e25fd48002d0c not found: ID does not exist" Nov 01 16:34:03.580737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.580555 4911 scope.go:122] "RemoveContainer" containerID="5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b" Nov 01 16:34:03.581946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.581922 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-2-w1-dfff59-kserve-59986f7bb7-kcs68"] Nov 01 16:34:03.590445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.590424 4911 scope.go:122] "RemoveContainer" containerID="5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b" Nov 01 16:34:03.590718 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:03.590698 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b\": container with ID starting with 5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b not found: ID does not exist" containerID="5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b" Nov 01 16:34:03.590767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.590726 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b"} err="failed to get container status \"5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b\": rpc error: code = NotFound desc = could not find container \"5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b\": container with ID starting with 5d4a6c9ca08add8734717c6b05e6edc4bcc239317e93632d2d76a35d1db0f04b not found: ID does not exist" Nov 01 16:34:03.596888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.596856 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h"] Nov 01 16:34:03.600444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.600421 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/simulator-w1-dfff59-kserve-8cf5d89c6-wx48h"] Nov 01 16:34:03.610968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.610918 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc"] Nov 01 16:34:03.614117 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.614091 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/premium-w1-dfff59-kserve-6c46b6d789-gs5jc"] Nov 01 16:34:03.966942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.966898 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" path="/var/lib/kubelet/pods/170897de-1a22-4f8f-be3b-d4b1b4322663/volumes" Nov 01 16:34:03.967464 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.967442 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9153161a-1192-4086-b3c5-8dcced55123f" path="/var/lib/kubelet/pods/9153161a-1192-4086-b3c5-8dcced55123f/volumes" Nov 01 16:34:03.967929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.967908 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" path="/var/lib/kubelet/pods/99a13fd4-ab3e-42a5-872e-8eccecef961c/volumes" Nov 01 16:34:03.968681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:03.968660 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" path="/var/lib/kubelet/pods/e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e/volumes" Nov 01 16:34:06.646735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:06.646692 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" containerID="cri-o://d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639" gracePeriod=45 Nov 01 16:34:06.799644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:06.799595 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused" Nov 01 16:34:06.923749 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:06.923713 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:34:07.017242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.017194 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tls-certs\") pod \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " Nov 01 16:34:07.017436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.017346 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-model-cache\") pod \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " Nov 01 16:34:07.017510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.017445 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-home\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-home\") pod \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " Nov 01 16:34:07.017510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.017488 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca8e5b5-b21b-4719-966a-e48aac6919ad-kube-api-access-mr82f\" (UniqueName: \"kubernetes.io/projected/2ca8e5b5-b21b-4719-966a-e48aac6919ad-kube-api-access-mr82f\") pod \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " Nov 01 16:34:07.017615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.017522 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-dshm\") pod \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " Nov 01 16:34:07.017615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.017551 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tmp-dir\") pod \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\" (UID: \"2ca8e5b5-b21b-4719-966a-e48aac6919ad\") " Nov 01 16:34:07.020000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.019965 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tmp-dir" pod "2ca8e5b5-b21b-4719-966a-e48aac6919ad" (UID: "2ca8e5b5-b21b-4719-966a-e48aac6919ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.020778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.020703 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-model-cache" pod "2ca8e5b5-b21b-4719-966a-e48aac6919ad" (UID: "2ca8e5b5-b21b-4719-966a-e48aac6919ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.020778 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.020761 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-home" pod "2ca8e5b5-b21b-4719-966a-e48aac6919ad" (UID: "2ca8e5b5-b21b-4719-966a-e48aac6919ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.021390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.021359 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca8e5b5-b21b-4719-966a-e48aac6919ad-kube-api-access-mr82f" pod "2ca8e5b5-b21b-4719-966a-e48aac6919ad" (UID: "2ca8e5b5-b21b-4719-966a-e48aac6919ad"). InnerVolumeSpecName "kube-api-access-mr82f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:07.021873 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.021833 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tls-certs" pod "2ca8e5b5-b21b-4719-966a-e48aac6919ad" (UID: "2ca8e5b5-b21b-4719-966a-e48aac6919ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:07.022051 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.022017 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-dshm" pod "2ca8e5b5-b21b-4719-966a-e48aac6919ad" (UID: "2ca8e5b5-b21b-4719-966a-e48aac6919ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:07.119190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.119152 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.119190 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.119192 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr82f\" (UniqueName: \"kubernetes.io/projected/2ca8e5b5-b21b-4719-966a-e48aac6919ad-kube-api-access-mr82f\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.119407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.119207 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.119407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.119217 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.119407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.119229 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8e5b5-b21b-4719-966a-e48aac6919ad-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.119407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.119237 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca8e5b5-b21b-4719-966a-e48aac6919ad-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.582913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.582873 4911 generic.go:356] "Generic (PLEG): container finished" podID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerID="d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639" exitCode=0 Nov 01 16:34:07.583260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.582923 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" Nov 01 16:34:07.583260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.582953 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" event={"ID":"2ca8e5b5-b21b-4719-966a-e48aac6919ad","Type":"ContainerDied","Data":"d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639"} Nov 01 16:34:07.583260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.583003 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5" event={"ID":"2ca8e5b5-b21b-4719-966a-e48aac6919ad","Type":"ContainerDied","Data":"8132a8361798019b686588f9c0b046093a4e7f6af61fb23b70892b94ca3d5aa0"} Nov 01 16:34:07.583260 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.583024 4911 scope.go:122] "RemoveContainer" containerID="d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639" Nov 01 16:34:07.594849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.594823 4911 scope.go:122] "RemoveContainer" containerID="d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639" Nov 01 16:34:07.595193 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:07.595171 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639\": container with ID starting with d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639 not found: ID does not exist" containerID="d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639" Nov 01 16:34:07.595267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.595203 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639"} err="failed to get container status \"d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639\": rpc error: code = NotFound desc = could not find container \"d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639\": container with ID starting with d395c2a113b48bfeacc17d73c3a8614b3b004b5ec645b19658ce3987dc244639 not found: ID does not exist" Nov 01 16:34:07.611309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.611118 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5"] Nov 01 16:34:07.616708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.616677 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-w4-9f4dc9-kserve-577bc6974-ztqc5"] Nov 01 16:34:07.968506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:07.968414 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" path="/var/lib/kubelet/pods/2ca8e5b5-b21b-4719-966a-e48aac6919ad/volumes" Nov 01 16:34:09.495921 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:09.495880 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg"] Nov 01 16:34:09.501862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:09.501801 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" podUID="56cead9c-dd49-4156-9505-0c0d4d195843" containerName="maas-api" containerID="cri-o://5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911" gracePeriod=30 Nov 01 16:34:10.094450 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.094417 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 16:34:10.205954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.205871 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cead9c-dd49-4156-9505-0c0d4d195843-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cead9c-dd49-4156-9505-0c0d4d195843-openshift-service-ca\") pod \"56cead9c-dd49-4156-9505-0c0d4d195843\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " Nov 01 16:34:10.205954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.205928 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-metrics-certs\") pod \"56cead9c-dd49-4156-9505-0c0d4d195843\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " Nov 01 16:34:10.205954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.205946 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-maas-api-tls\") pod \"56cead9c-dd49-4156-9505-0c0d4d195843\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " Nov 01 16:34:10.206305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.206058 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cead9c-dd49-4156-9505-0c0d4d195843-kube-api-access-sr7vm\" (UniqueName: \"kubernetes.io/projected/56cead9c-dd49-4156-9505-0c0d4d195843-kube-api-access-sr7vm\") pod \"56cead9c-dd49-4156-9505-0c0d4d195843\" (UID: \"56cead9c-dd49-4156-9505-0c0d4d195843\") " Nov 01 16:34:10.206532 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.206498 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cead9c-dd49-4156-9505-0c0d4d195843-openshift-service-ca" pod "56cead9c-dd49-4156-9505-0c0d4d195843" (UID: "56cead9c-dd49-4156-9505-0c0d4d195843"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:10.208792 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.208760 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-maas-api-tls" pod "56cead9c-dd49-4156-9505-0c0d4d195843" (UID: "56cead9c-dd49-4156-9505-0c0d4d195843"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:10.208792 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.208779 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-metrics-certs" pod "56cead9c-dd49-4156-9505-0c0d4d195843" (UID: "56cead9c-dd49-4156-9505-0c0d4d195843"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:10.208950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.208781 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cead9c-dd49-4156-9505-0c0d4d195843-kube-api-access-sr7vm" pod "56cead9c-dd49-4156-9505-0c0d4d195843" (UID: "56cead9c-dd49-4156-9505-0c0d4d195843"). InnerVolumeSpecName "kube-api-access-sr7vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:10.307626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.307583 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cead9c-dd49-4156-9505-0c0d4d195843-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:10.307626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.307627 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:10.307915 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.307645 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cead9c-dd49-4156-9505-0c0d4d195843-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:10.307915 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.307659 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr7vm\" (UniqueName: \"kubernetes.io/projected/56cead9c-dd49-4156-9505-0c0d4d195843-kube-api-access-sr7vm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:10.634067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.631910 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl"] Nov 01 16:34:10.634067 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.632167 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" podUID="1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" containerName="maas-api" containerID="cri-o://e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07" gracePeriod=30 Nov 01 16:34:10.658807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.658770 4911 generic.go:356] "Generic (PLEG): container finished" podID="56cead9c-dd49-4156-9505-0c0d4d195843" containerID="5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911" exitCode=0 Nov 01 16:34:10.659005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.658838 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" Nov 01 16:34:10.659005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.658910 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" event={"ID":"56cead9c-dd49-4156-9505-0c0d4d195843","Type":"ContainerDied","Data":"5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911"} Nov 01 16:34:10.659005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.658953 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg" event={"ID":"56cead9c-dd49-4156-9505-0c0d4d195843","Type":"ContainerDied","Data":"faafdca2120f26ca770f352e676eba938ec38b349c03dbfd35a2bc3a5f2fa336"} Nov 01 16:34:10.659005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.658985 4911 scope.go:122] "RemoveContainer" containerID="5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911" Nov 01 16:34:10.732683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.732652 4911 scope.go:122] "RemoveContainer" containerID="5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911" Nov 01 16:34:10.733180 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:10.733142 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911\": container with ID starting with 5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911 not found: ID does not exist" containerID="5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911" Nov 01 16:34:10.733339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.733189 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911"} err="failed to get container status \"5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911\": rpc error: code = NotFound desc = could not find container \"5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911\": container with ID starting with 5e3b4c8697b2ac10d46e35b5edea1026d557096dcf23fd40456a1b1d5c973911 not found: ID does not exist" Nov 01 16:34:10.744381 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.744345 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg"] Nov 01 16:34:10.751314 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.751284 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dfff59-6c5966d7f6-ls8fg"] Nov 01 16:34:10.805205 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.805055 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" podUID="1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": dial tcp 10.132.0.66:8443: connect: connection refused" Nov 01 16:34:10.914937 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:10.914872 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:34:10.991096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:10.991068 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 16:34:11.117340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.117300 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-metrics-certs\") pod \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " Nov 01 16:34:11.117546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.117375 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-kube-api-access-hmchs\" (UniqueName: \"kubernetes.io/projected/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-kube-api-access-hmchs\") pod \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " Nov 01 16:34:11.117546 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.117456 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls\") pod \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " Nov 01 16:34:11.117682 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.117600 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-openshift-service-ca\") pod \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\" (UID: \"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d\") " Nov 01 16:34:11.117978 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.117946 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-openshift-service-ca" pod "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" (UID: "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:11.119608 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.119575 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-kube-api-access-hmchs" pod "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" (UID: "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d"). InnerVolumeSpecName "kube-api-access-hmchs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:11.119608 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.119580 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-metrics-certs" pod "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" (UID: "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:11.119777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.119607 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls" pod "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" (UID: "1c7dbfa6-cea0-4fd7-ae29-471160afbd7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:11.219008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.218926 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmchs\" (UniqueName: \"kubernetes.io/projected/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-kube-api-access-hmchs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:11.219008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.218960 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:11.219008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.218970 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:11.219008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.218979 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:11.666869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.666836 4911 generic.go:356] "Generic (PLEG): container finished" podID="1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" containerID="e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07" exitCode=0 Nov 01 16:34:11.667343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.666882 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" Nov 01 16:34:11.667343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.666930 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" event={"ID":"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d","Type":"ContainerDied","Data":"e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07"} Nov 01 16:34:11.667343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.666978 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl" event={"ID":"1c7dbfa6-cea0-4fd7-ae29-471160afbd7d","Type":"ContainerDied","Data":"87cea3b99d06f54061863f72a02d2251fe5eda903211bee40b2dcf21ebcc13e2"} Nov 01 16:34:11.667343 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.667003 4911 scope.go:122] "RemoveContainer" containerID="e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07" Nov 01 16:34:11.678253 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.678231 4911 scope.go:122] "RemoveContainer" containerID="e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07" Nov 01 16:34:11.678536 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:11.678513 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07\": container with ID starting with e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07 not found: ID does not exist" containerID="e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07" Nov 01 16:34:11.678583 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.678545 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07"} err="failed to get container status \"e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07\": rpc error: code = NotFound desc = could not find container \"e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07\": container with ID starting with e28c80082a23504c500029d7d1e304640abcadec37953113d4f2c8f2c8389f07 not found: ID does not exist" Nov 01 16:34:11.710895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.710862 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl"] Nov 01 16:34:11.720707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.720679 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9f4dc9-7795b4f98b-n6jjl"] Nov 01 16:34:11.966244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.966162 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" path="/var/lib/kubelet/pods/1c7dbfa6-cea0-4fd7-ae29-471160afbd7d/volumes" Nov 01 16:34:11.966692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:11.966671 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56cead9c-dd49-4156-9505-0c0d4d195843" path="/var/lib/kubelet/pods/56cead9c-dd49-4156-9505-0c0d4d195843/volumes" Nov 01 16:34:13.840019 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.839978 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw"] Nov 01 16:34:13.840467 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840452 4911 cpu_manager.go:414] "containerMap: removing container" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" Nov 01 16:34:13.840513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840468 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" Nov 01 16:34:13.840513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840479 4911 cpu_manager.go:414] "containerMap: removing container" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" Nov 01 16:34:13.840513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840485 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" Nov 01 16:34:13.840513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840492 4911 cpu_manager.go:414] "containerMap: removing container" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" Nov 01 16:34:13.840513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840498 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" Nov 01 16:34:13.840513 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840511 4911 cpu_manager.go:414] "containerMap: removing container" podUID="c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" containerName="revoke-keys" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840517 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" containerName="revoke-keys" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840530 4911 cpu_manager.go:414] "containerMap: removing container" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840534 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840541 4911 cpu_manager.go:414] "containerMap: removing container" podUID="56cead9c-dd49-4156-9505-0c0d4d195843" containerName="maas-api" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840545 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56cead9c-dd49-4156-9505-0c0d4d195843" containerName="maas-api" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840554 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" containerName="maas-api" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840558 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" containerName="maas-api" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840569 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" containerName="payload-processing" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840574 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" containerName="payload-processing" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840650 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca8e5b5-b21b-4719-966a-e48aac6919ad" containerName="main" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840658 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="9153161a-1192-4086-b3c5-8dcced55123f" containerName="main" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840664 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="170897de-1a22-4f8f-be3b-d4b1b4322663" containerName="main" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840670 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" containerName="revoke-keys" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840676 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a13fd4-ab3e-42a5-872e-8eccecef961c" containerName="main" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840681 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="56cead9c-dd49-4156-9505-0c0d4d195843" containerName="maas-api" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840689 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7dbfa6-cea0-4fd7-ae29-471160afbd7d" containerName="maas-api" Nov 01 16:34:13.840706 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.840695 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ea7536-d3c6-4425-89c1-5ba1a36eaa7e" containerName="payload-processing" Nov 01 16:34:13.845500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.845479 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:13.850754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.850730 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:34:13.850955 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.850886 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:34:13.851157 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.851076 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw"] Nov 01 16:34:13.955722 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.955675 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:13.955937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.955736 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:13.955937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:13.955897 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjfbw\" (UniqueName: \"kubernetes.io/projected/e0d77235-805f-4dd5-89ee-9a3676f29cc1-kube-api-access-gjfbw\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.057249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.057207 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.057249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.057253 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.057566 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.057328 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfbw\" (UniqueName: \"kubernetes.io/projected/e0d77235-805f-4dd5-89ee-9a3676f29cc1-kube-api-access-gjfbw\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.057907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.057880 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.059816 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.059786 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.066570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.066544 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjfbw\" (UniqueName: \"kubernetes.io/projected/e0d77235-805f-4dd5-89ee-9a3676f29cc1-kube-api-access-gjfbw\") pod \"maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.164126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.164017 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:14.460208 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:34:14.456416 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d77235_805f_4dd5_89ee_9a3676f29cc1.slice/crio-45431d80f6c7a86bd2b498f1125d05a024859972f82866b312895ca4e8c40e9d WatchSource:0}: Error finding container 45431d80f6c7a86bd2b498f1125d05a024859972f82866b312895ca4e8c40e9d: Status 404 returned error can't find the container with id 45431d80f6c7a86bd2b498f1125d05a024859972f82866b312895ca4e8c40e9d Nov 01 16:34:14.462928 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.462892 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw"] Nov 01 16:34:14.691382 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:14.691334 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" event={"ID":"e0d77235-805f-4dd5-89ee-9a3676f29cc1","Type":"ContainerStarted","Data":"45431d80f6c7a86bd2b498f1125d05a024859972f82866b312895ca4e8c40e9d"} Nov 01 16:34:15.701366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:15.701328 4911 generic.go:356] "Generic (PLEG): container finished" podID="e0d77235-805f-4dd5-89ee-9a3676f29cc1" containerID="f76ba69347b601c3891ab3e6060a718706e5a98a68144b67ee8f6fe9db1f09bc" exitCode=0 Nov 01 16:34:15.701771 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:15.701411 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" event={"ID":"e0d77235-805f-4dd5-89ee-9a3676f29cc1","Type":"ContainerDied","Data":"f76ba69347b601c3891ab3e6060a718706e5a98a68144b67ee8f6fe9db1f09bc"} Nov 01 16:34:16.852953 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:16.852919 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:16.993492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:16.993461 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-xks-ca\") pod \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " Nov 01 16:34:16.993689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:16.993527 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-openshift-ca\") pod \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " Nov 01 16:34:16.993689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:16.993569 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d77235-805f-4dd5-89ee-9a3676f29cc1-kube-api-access-gjfbw\" (UniqueName: \"kubernetes.io/projected/e0d77235-805f-4dd5-89ee-9a3676f29cc1-kube-api-access-gjfbw\") pod \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\" (UID: \"e0d77235-805f-4dd5-89ee-9a3676f29cc1\") " Nov 01 16:34:16.993954 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:16.993930 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-openshift-ca" pod "e0d77235-805f-4dd5-89ee-9a3676f29cc1" (UID: "e0d77235-805f-4dd5-89ee-9a3676f29cc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:16.995824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:16.995800 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-xks-ca" pod "e0d77235-805f-4dd5-89ee-9a3676f29cc1" (UID: "e0d77235-805f-4dd5-89ee-9a3676f29cc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:16.995910 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:16.995821 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d77235-805f-4dd5-89ee-9a3676f29cc1-kube-api-access-gjfbw" pod "e0d77235-805f-4dd5-89ee-9a3676f29cc1" (UID: "e0d77235-805f-4dd5-89ee-9a3676f29cc1"). InnerVolumeSpecName "kube-api-access-gjfbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:17.094693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.094653 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:17.094693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.094684 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d77235-805f-4dd5-89ee-9a3676f29cc1-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:17.094693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.094695 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjfbw\" (UniqueName: \"kubernetes.io/projected/e0d77235-805f-4dd5-89ee-9a3676f29cc1-kube-api-access-gjfbw\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:17.404151 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.404061 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q"] Nov 01 16:34:17.404659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.404601 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" containerID="cri-o://eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097" gracePeriod=30 Nov 01 16:34:17.716423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.716336 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" event={"ID":"e0d77235-805f-4dd5-89ee-9a3676f29cc1","Type":"ContainerDied","Data":"45431d80f6c7a86bd2b498f1125d05a024859972f82866b312895ca4e8c40e9d"} Nov 01 16:34:17.716423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.716392 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45431d80f6c7a86bd2b498f1125d05a024859972f82866b312895ca4e8c40e9d" Nov 01 16:34:17.716423 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:17.716348 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw" Nov 01 16:34:21.343083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:21.343017 4911 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:34:21.343509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:21.343110 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:34:22.680496 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.680428 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:34:22.747879 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.747839 4911 generic.go:356] "Generic (PLEG): container finished" podID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerID="eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097" exitCode=0 Nov 01 16:34:22.748085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.747892 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" Nov 01 16:34:22.748085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.747916 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" event={"ID":"f53180dc-7fc1-417d-93e2-23dfc9ea3673","Type":"ContainerDied","Data":"eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097"} Nov 01 16:34:22.748085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.747974 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q" event={"ID":"f53180dc-7fc1-417d-93e2-23dfc9ea3673","Type":"ContainerDied","Data":"ffabe99859cb6a8dc12a33b3ae8bba05218eaa5ea11a77b27468f4f161b6aca1"} Nov 01 16:34:22.748085 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.747996 4911 scope.go:122] "RemoveContainer" containerID="eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097" Nov 01 16:34:22.756848 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.756823 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-envoy\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.756974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.756926 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istiod-ca-cert\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.756974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.756968 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-data\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.757097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.756996 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-podinfo\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.757310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.757250 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-data" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.757410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.757369 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istiod-ca-cert" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:22.757410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.757379 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.758999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.758898 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-envoy" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.759132 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.759108 4911 scope.go:122] "RemoveContainer" containerID="eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097" Nov 01 16:34:22.759220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.759171 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-podinfo" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:34:22.759427 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:22.759406 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097\": container with ID starting with eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097 not found: ID does not exist" containerID="eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097" Nov 01 16:34:22.759497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.759438 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097"} err="failed to get container status \"eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097\": rpc error: code = NotFound desc = could not find container \"eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097\": container with ID starting with eb001fe8c4cf39a4231085abb9f45026d2f3162537e80462d0945351038ad097 not found: ID does not exist" Nov 01 16:34:22.858346 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858308 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-token\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-token\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.858564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858366 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-certs\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.858564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858393 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-socket\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.858564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858415 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-credential-socket\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.858564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858550 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-kube-api-access-dg4qv\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-kube-api-access-dg4qv\") pod \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\" (UID: \"f53180dc-7fc1-417d-93e2-23dfc9ea3673\") " Nov 01 16:34:22.858798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858694 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-socket" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.858798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858704 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-credential-socket" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.858798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.858719 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-certs" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.859394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.859367 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.859520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.859399 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.859520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.859415 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.859520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.859431 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.859520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.859447 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.859520 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.859462 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.860735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.860711 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-kube-api-access-dg4qv" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "kube-api-access-dg4qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:22.860870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.860850 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-token" pod "f53180dc-7fc1-417d-93e2-23dfc9ea3673" (UID: "f53180dc-7fc1-417d-93e2-23dfc9ea3673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:22.960750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.960664 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.960750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:22.960697 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg4qv\" (UniqueName: \"kubernetes.io/projected/f53180dc-7fc1-417d-93e2-23dfc9ea3673-kube-api-access-dg4qv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:23.072605 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:23.072560 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q"] Nov 01 16:34:23.077765 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:23.077729 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5gq9q"] Nov 01 16:34:23.965543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:23.965510 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" path="/var/lib/kubelet/pods/f53180dc-7fc1-417d-93e2-23dfc9ea3673/volumes" Nov 01 16:34:24.504713 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:24.504664 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw"] Nov 01 16:34:24.507152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:24.507121 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-501d63-j49dw"] Nov 01 16:34:25.976183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:34:25.976137 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d77235-805f-4dd5-89ee-9a3676f29cc1" path="/var/lib/kubelet/pods/e0d77235-805f-4dd5-89ee-9a3676f29cc1/volumes" Nov 01 16:34:40.866830 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:34:40.866791 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:35:10.415876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:10.415829 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4"] Nov 01 16:35:10.416493 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:10.416165 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" containerID="cri-o://5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e" gracePeriod=30 Nov 01 16:35:10.871936 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:10.871893 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:35:15.677888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.677862 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:35:15.769803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.769764 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-certs\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.769856 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b9a8165-4040-46b2-b600-e423d00f851c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9a8165-4040-46b2-b600-e423d00f851c-istio-podinfo\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.769907 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9a8165-4040-46b2-b600-e423d00f851c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9a8165-4040-46b2-b600-e423d00f851c-istiod-ca-cert\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.769928 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-socket\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770093 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-istio-token\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-istio-token\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770104 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-certs" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.770238 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770134 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-envoy\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770261 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-kube-api-access-kqn6t\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-kube-api-access-kqn6t\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770274 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-socket" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.770419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770329 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-data\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770419 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770363 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-credential-socket\") pod \"5b9a8165-4040-46b2-b600-e423d00f851c\" (UID: \"5b9a8165-4040-46b2-b600-e423d00f851c\") " Nov 01 16:35:15.770641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770345 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9a8165-4040-46b2-b600-e423d00f851c-istiod-ca-cert" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:15.770641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770621 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-data" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.770841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770816 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.770912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770844 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9a8165-4040-46b2-b600-e423d00f851c-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.770912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770860 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.770912 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770874 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.771123 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.770938 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-credential-socket" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.772504 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.772470 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b9a8165-4040-46b2-b600-e423d00f851c-istio-podinfo" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:15.772629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.772474 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-envoy" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.772629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.772532 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-istio-token" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.772629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.772572 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-kube-api-access-kqn6t" pod "5b9a8165-4040-46b2-b600-e423d00f851c" (UID: "5b9a8165-4040-46b2-b600-e423d00f851c"). InnerVolumeSpecName "kube-api-access-kqn6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.872453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.872413 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9a8165-4040-46b2-b600-e423d00f851c-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.872453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.872445 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.872453 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.872455 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.872709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.872464 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqn6t\" (UniqueName: \"kubernetes.io/projected/5b9a8165-4040-46b2-b600-e423d00f851c-kube-api-access-kqn6t\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.872709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:15.872474 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9a8165-4040-46b2-b600-e423d00f851c-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:16.079800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.079710 4911 generic.go:356] "Generic (PLEG): container finished" podID="5b9a8165-4040-46b2-b600-e423d00f851c" containerID="5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e" exitCode=0 Nov 01 16:35:16.079800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.079776 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" Nov 01 16:35:16.080016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.079795 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" event={"ID":"5b9a8165-4040-46b2-b600-e423d00f851c","Type":"ContainerDied","Data":"5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e"} Nov 01 16:35:16.080016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.079853 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4" event={"ID":"5b9a8165-4040-46b2-b600-e423d00f851c","Type":"ContainerDied","Data":"24609746c9b4b1cbecbbde7c8ef892b347408044c0b07751669efd8bad0e8954"} Nov 01 16:35:16.080016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.079877 4911 scope.go:122] "RemoveContainer" containerID="5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e" Nov 01 16:35:16.090276 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.090255 4911 scope.go:122] "RemoveContainer" containerID="5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e" Nov 01 16:35:16.090560 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:16.090537 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e\": container with ID starting with 5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e not found: ID does not exist" containerID="5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e" Nov 01 16:35:16.090629 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.090572 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e"} err="failed to get container status \"5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e\": rpc error: code = NotFound desc = could not find container \"5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e\": container with ID starting with 5fb525ccb2d45b040619dcef49f9549ad9bbed439da8d0c9f07909b6d8a0918e not found: ID does not exist" Nov 01 16:35:16.098359 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.098333 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4"] Nov 01 16:35:16.103973 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:16.103946 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-nv6w4"] Nov 01 16:35:17.968458 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:17.968422 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" path="/var/lib/kubelet/pods/5b9a8165-4040-46b2-b600-e423d00f851c/volumes" Nov 01 16:35:19.026071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:19.026017 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7"] Nov 01 16:35:19.028441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:19.028411 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d35e1bb7-wzwt7"] Nov 01 16:35:19.965968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:19.965932 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e436d9-9dde-41f3-99d7-4086a17530df" path="/var/lib/kubelet/pods/d5e436d9-9dde-41f3-99d7-4086a17530df/volumes" Nov 01 16:35:22.452929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.452882 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq"] Nov 01 16:35:22.453570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.453237 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" podUID="498dc930-9369-454f-83bd-b61c60db83ad" containerName="maas-api" containerID="cri-o://274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef" gracePeriod=30 Nov 01 16:35:22.767938 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.767913 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:35:22.946528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.946496 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498dc930-9369-454f-83bd-b61c60db83ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/498dc930-9369-454f-83bd-b61c60db83ad-openshift-service-ca\") pod \"498dc930-9369-454f-83bd-b61c60db83ad\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " Nov 01 16:35:22.946739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.946541 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-maas-api-tls\") pod \"498dc930-9369-454f-83bd-b61c60db83ad\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " Nov 01 16:35:22.946739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.946692 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498dc930-9369-454f-83bd-b61c60db83ad-kube-api-access-hxml4\" (UniqueName: \"kubernetes.io/projected/498dc930-9369-454f-83bd-b61c60db83ad-kube-api-access-hxml4\") pod \"498dc930-9369-454f-83bd-b61c60db83ad\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " Nov 01 16:35:22.946739 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.946735 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-metrics-certs\") pod \"498dc930-9369-454f-83bd-b61c60db83ad\" (UID: \"498dc930-9369-454f-83bd-b61c60db83ad\") " Nov 01 16:35:22.946972 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.946949 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498dc930-9369-454f-83bd-b61c60db83ad-openshift-service-ca" pod "498dc930-9369-454f-83bd-b61c60db83ad" (UID: "498dc930-9369-454f-83bd-b61c60db83ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:22.947183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.947149 4911 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/498dc930-9369-454f-83bd-b61c60db83ad-openshift-service-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:22.948910 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.948873 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-metrics-certs" pod "498dc930-9369-454f-83bd-b61c60db83ad" (UID: "498dc930-9369-454f-83bd-b61c60db83ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:22.949073 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.948935 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498dc930-9369-454f-83bd-b61c60db83ad-kube-api-access-hxml4" pod "498dc930-9369-454f-83bd-b61c60db83ad" (UID: "498dc930-9369-454f-83bd-b61c60db83ad"). InnerVolumeSpecName "kube-api-access-hxml4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:22.949150 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:22.949091 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-maas-api-tls" pod "498dc930-9369-454f-83bd-b61c60db83ad" (UID: "498dc930-9369-454f-83bd-b61c60db83ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:23.048361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.048255 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-maas-api-tls\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:23.048361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.048292 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxml4\" (UniqueName: \"kubernetes.io/projected/498dc930-9369-454f-83bd-b61c60db83ad-kube-api-access-hxml4\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:23.048361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.048303 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498dc930-9369-454f-83bd-b61c60db83ad-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:23.123071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.123021 4911 generic.go:356] "Generic (PLEG): container finished" podID="498dc930-9369-454f-83bd-b61c60db83ad" containerID="274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef" exitCode=0 Nov 01 16:35:23.123252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.123078 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" Nov 01 16:35:23.123252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.123072 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" event={"ID":"498dc930-9369-454f-83bd-b61c60db83ad","Type":"ContainerDied","Data":"274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef"} Nov 01 16:35:23.123252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.123200 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq" event={"ID":"498dc930-9369-454f-83bd-b61c60db83ad","Type":"ContainerDied","Data":"556239102a99a89b10de002a1d7386f2a6c6d72521ed960e6b14ed37fc0eb7fe"} Nov 01 16:35:23.123252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.123213 4911 scope.go:122] "RemoveContainer" containerID="274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef" Nov 01 16:35:23.134185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.134167 4911 scope.go:122] "RemoveContainer" containerID="274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef" Nov 01 16:35:23.134434 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:23.134411 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef\": container with ID starting with 274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef not found: ID does not exist" containerID="274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef" Nov 01 16:35:23.134476 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.134443 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef"} err="failed to get container status \"274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef\": rpc error: code = NotFound desc = could not find container \"274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef\": container with ID starting with 274fea64303256f883c239fee50a5f225b84d8b5d6d16e7abfcd3639957085ef not found: ID does not exist" Nov 01 16:35:23.146609 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.146586 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq"] Nov 01 16:35:23.149936 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.149911 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a9a967-7f84758579-flxcq"] Nov 01 16:35:23.965811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:23.965762 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498dc930-9369-454f-83bd-b61c60db83ad" path="/var/lib/kubelet/pods/498dc930-9369-454f-83bd-b61c60db83ad/volumes" Nov 01 16:35:29.754410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:29.754373 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv"] Nov 01 16:35:29.754897 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:29.754634 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" containerID="cri-o://ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22" gracePeriod=30 Nov 01 16:35:35.018957 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.018928 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:35:35.060068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.059967 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-envoy\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060068 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060018 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-certs\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060084 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-credential-socket\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060147 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-data\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060185 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istiod-ca-cert\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060242 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-socket\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060321 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060301 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-kube-api-access-5q9v8\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-kube-api-access-5q9v8\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060339 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-podinfo\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060683 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060389 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-token\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-token\") pod \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\" (UID: \"5e6e888f-bcad-46f7-8c32-04b78cfedd6b\") " Nov 01 16:35:35.060934 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060334 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-certs" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.060934 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060922 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-data" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.061109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060628 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-credential-socket" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.061109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.060657 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istiod-ca-cert" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:35.061466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.061441 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-socket" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.062774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.062743 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-envoy" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.063217 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.063194 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-podinfo" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:35.063303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.063261 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-token" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.063720 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.063692 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-kube-api-access-5q9v8" pod "5e6e888f-bcad-46f7-8c32-04b78cfedd6b" (UID: "5e6e888f-bcad-46f7-8c32-04b78cfedd6b"). InnerVolumeSpecName "kube-api-access-5q9v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.162137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162100 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162131 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162137 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162142 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162151 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162161 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162169 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q9v8\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-kube-api-access-5q9v8\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162179 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162187 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.162388 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.162195 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6e888f-bcad-46f7-8c32-04b78cfedd6b-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.191164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.191131 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97"] Nov 01 16:35:35.195406 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.195374 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg"] Nov 01 16:35:35.198023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.197979 4911 generic.go:356] "Generic (PLEG): container finished" podID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerID="ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22" exitCode=0 Nov 01 16:35:35.198198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.198025 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" event={"ID":"5e6e888f-bcad-46f7-8c32-04b78cfedd6b","Type":"ContainerDied","Data":"ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22"} Nov 01 16:35:35.198198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.198118 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" event={"ID":"5e6e888f-bcad-46f7-8c32-04b78cfedd6b","Type":"ContainerDied","Data":"4ad8d7926e536b11c4f49e0241262f4676fa78924fc48eefae3d2be1ddc34f30"} Nov 01 16:35:35.198198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.198033 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv" Nov 01 16:35:35.198387 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.198150 4911 scope.go:122] "RemoveContainer" containerID="ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22" Nov 01 16:35:35.212911 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.212876 4911 scope.go:122] "RemoveContainer" containerID="ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22" Nov 01 16:35:35.213325 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:35.213296 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22\": container with ID starting with ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22 not found: ID does not exist" containerID="ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22" Nov 01 16:35:35.213425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.213337 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22"} err="failed to get container status \"ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22\": rpc error: code = NotFound desc = could not find container \"ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22\": container with ID starting with ba785db5af189bbbdc340f34e98fd5dc5fb292356f1a2215c500b93943f5ad22 not found: ID does not exist" Nov 01 16:35:35.213837 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.213823 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8"] Nov 01 16:35:35.250378 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.250338 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv"] Nov 01 16:35:35.257768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.257733 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-vdmvv"] Nov 01 16:35:35.965641 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:35.965609 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" path="/var/lib/kubelet/pods/5e6e888f-bcad-46f7-8c32-04b78cfedd6b/volumes" Nov 01 16:35:40.474262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.474221 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6"] Nov 01 16:35:40.867842 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:40.867801 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:35:40.915313 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915278 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq"] Nov 01 16:35:40.915777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915757 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" Nov 01 16:35:40.915869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915778 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" Nov 01 16:35:40.915869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915796 4911 cpu_manager.go:414] "containerMap: removing container" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" Nov 01 16:35:40.915869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915804 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" Nov 01 16:35:40.915869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915842 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e0d77235-805f-4dd5-89ee-9a3676f29cc1" containerName="revoke-keys" Nov 01 16:35:40.915869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915850 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d77235-805f-4dd5-89ee-9a3676f29cc1" containerName="revoke-keys" Nov 01 16:35:40.915869 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915869 4911 cpu_manager.go:414] "containerMap: removing container" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915877 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915894 4911 cpu_manager.go:414] "containerMap: removing container" podUID="498dc930-9369-454f-83bd-b61c60db83ad" containerName="maas-api" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.915902 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498dc930-9369-454f-83bd-b61c60db83ad" containerName="maas-api" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.916013 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="498dc930-9369-454f-83bd-b61c60db83ad" containerName="maas-api" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.916029 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53180dc-7fc1-417d-93e2-23dfc9ea3673" containerName="istio-proxy" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.916055 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9a8165-4040-46b2-b600-e423d00f851c" containerName="istio-proxy" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.916074 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d77235-805f-4dd5-89ee-9a3676f29cc1" containerName="revoke-keys" Nov 01 16:35:40.916216 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.916084 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6e888f-bcad-46f7-8c32-04b78cfedd6b" containerName="istio-proxy" Nov 01 16:35:40.919403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.919381 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:40.925634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.925604 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:35:40.928448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.927714 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:35:40.928448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.927985 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:35:40.928448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.928416 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:35:40.931817 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:40.931788 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq"] Nov 01 16:35:41.021349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.021299 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.021349 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.021352 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.021625 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.021448 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jjz\" (UniqueName: \"kubernetes.io/projected/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-kube-api-access-r6jjz\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.122665 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.122571 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.122665 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.122620 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.122665 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.122653 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jjz\" (UniqueName: \"kubernetes.io/projected/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-kube-api-access-r6jjz\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.123424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.123402 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.125002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.124984 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.131616 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.131589 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jjz\" (UniqueName: \"kubernetes.io/projected/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-kube-api-access-r6jjz\") pod \"maas-api-revoke-sub-dd4960a2-pfgsq\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.238583 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.238553 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:41.519247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:41.519216 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq"] Nov 01 16:35:41.520846 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:35:41.520817 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf98d2fb_1d3e_4c66_8644_f80bc5e0a16b.slice/crio-172df9b02622016c323524bb1867f7e3e3b10584054c918981ebfba5cda641a3 WatchSource:0}: Error finding container 172df9b02622016c323524bb1867f7e3e3b10584054c918981ebfba5cda641a3: Status 404 returned error can't find the container with id 172df9b02622016c323524bb1867f7e3e3b10584054c918981ebfba5cda641a3 Nov 01 16:35:42.241793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:42.241757 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" event={"ID":"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b","Type":"ContainerStarted","Data":"1443e2bcb41fee671febc0e3952e8a56bcadd71acd32938489632b41988e3af1"} Nov 01 16:35:42.241969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:42.241809 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" event={"ID":"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b","Type":"ContainerStarted","Data":"172df9b02622016c323524bb1867f7e3e3b10584054c918981ebfba5cda641a3"} Nov 01 16:35:42.258841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:42.258794 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" podStartSLOduration=2.258780299 podStartE2EDuration="2.258780299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:35:42.257479055 +0000 UTC m=+2882.401933697" watchObservedRunningTime="2026-11-01 16:35:42.258780299 +0000 UTC m=+2882.403234983" Nov 01 16:35:43.251004 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:43.250959 4911 generic.go:356] "Generic (PLEG): container finished" podID="af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" containerID="1443e2bcb41fee671febc0e3952e8a56bcadd71acd32938489632b41988e3af1" exitCode=0 Nov 01 16:35:43.251659 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:43.251067 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" event={"ID":"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b","Type":"ContainerDied","Data":"1443e2bcb41fee671febc0e3952e8a56bcadd71acd32938489632b41988e3af1"} Nov 01 16:35:44.404998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.404969 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:44.455490 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.455457 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-kube-api-access-r6jjz\" (UniqueName: \"kubernetes.io/projected/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-kube-api-access-r6jjz\") pod \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " Nov 01 16:35:44.455681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.455501 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-xks-ca\") pod \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " Nov 01 16:35:44.455681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.455523 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-openshift-ca\") pod \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\" (UID: \"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b\") " Nov 01 16:35:44.455947 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.455918 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-openshift-ca" pod "af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" (UID: "af98d2fb-1d3e-4c66-8644-f80bc5e0a16b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:44.457752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.457724 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-xks-ca" pod "af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" (UID: "af98d2fb-1d3e-4c66-8644-f80bc5e0a16b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:44.457752 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.457730 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-kube-api-access-r6jjz" pod "af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" (UID: "af98d2fb-1d3e-4c66-8644-f80bc5e0a16b"). InnerVolumeSpecName "kube-api-access-r6jjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:44.556969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.556883 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6jjz\" (UniqueName: \"kubernetes.io/projected/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-kube-api-access-r6jjz\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:44.556969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.556915 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:44.556969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:44.556925 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:45.263979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:45.263941 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" Nov 01 16:35:45.263979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:45.263956 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq" event={"ID":"af98d2fb-1d3e-4c66-8644-f80bc5e0a16b","Type":"ContainerDied","Data":"172df9b02622016c323524bb1867f7e3e3b10584054c918981ebfba5cda641a3"} Nov 01 16:35:45.264213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:45.264000 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="172df9b02622016c323524bb1867f7e3e3b10584054c918981ebfba5cda641a3" Nov 01 16:35:45.475123 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:45.475082 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" podUID="3475b4c0-cc38-47d7-b344-8c2170eee46e" containerName="payload-pre-processing" containerID="cri-o://188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589" gracePeriod=25 Nov 01 16:35:50.218154 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.218115 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" containerID="cri-o://01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad" gracePeriod=45 Nov 01 16:35:50.220724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.220689 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" containerID="cri-o://3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617" gracePeriod=45 Nov 01 16:35:50.236478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.236443 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" containerID="cri-o://cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c" gracePeriod=45 Nov 01 16:35:50.361476 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.361435 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": dial tcp 10.132.0.67:8000: connect: connection refused" Nov 01 16:35:50.518181 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.518154 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:35:50.521681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.521647 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:35:50.565261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.565227 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:35:50.617758 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.617722 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-model-cache\") pod \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " Nov 01 16:35:50.617906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.617765 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-home\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-home\") pod \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " Nov 01 16:35:50.617906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.617814 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-home\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-home\") pod \"474d2b5a-53c8-43f8-8892-23f3d843febb\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " Nov 01 16:35:50.617906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.617852 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tls-certs\") pod \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " Nov 01 16:35:50.618112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.617902 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-kube-api-access-tlcsn\" (UniqueName: \"kubernetes.io/projected/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-kube-api-access-tlcsn\") pod \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " Nov 01 16:35:50.618112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.617934 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-model-cache\") pod \"474d2b5a-53c8-43f8-8892-23f3d843febb\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " Nov 01 16:35:50.618112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618017 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-dshm\") pod \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " Nov 01 16:35:50.618336 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618315 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-tmp-dir\") pod \"474d2b5a-53c8-43f8-8892-23f3d843febb\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " Nov 01 16:35:50.618405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618071 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-model-cache" pod "aba15e06-9a58-43e7-94c7-8d8aba22d4e5" (UID: "aba15e06-9a58-43e7-94c7-8d8aba22d4e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.618405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618353 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474d2b5a-53c8-43f8-8892-23f3d843febb-tls-certs\" (UniqueName: \"kubernetes.io/secret/474d2b5a-53c8-43f8-8892-23f3d843febb-tls-certs\") pod \"474d2b5a-53c8-43f8-8892-23f3d843febb\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " Nov 01 16:35:50.618405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618156 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-home" pod "474d2b5a-53c8-43f8-8892-23f3d843febb" (UID: "474d2b5a-53c8-43f8-8892-23f3d843febb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.618405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618179 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-home" pod "aba15e06-9a58-43e7-94c7-8d8aba22d4e5" (UID: "aba15e06-9a58-43e7-94c7-8d8aba22d4e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.618405 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618249 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-model-cache" pod "474d2b5a-53c8-43f8-8892-23f3d843febb" (UID: "474d2b5a-53c8-43f8-8892-23f3d843febb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.618693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618418 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-dshm\") pod \"474d2b5a-53c8-43f8-8892-23f3d843febb\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " Nov 01 16:35:50.618693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618471 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tmp-dir\") pod \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\" (UID: \"aba15e06-9a58-43e7-94c7-8d8aba22d4e5\") " Nov 01 16:35:50.618693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618530 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474d2b5a-53c8-43f8-8892-23f3d843febb-kube-api-access-422qm\" (UniqueName: \"kubernetes.io/projected/474d2b5a-53c8-43f8-8892-23f3d843febb-kube-api-access-422qm\") pod \"474d2b5a-53c8-43f8-8892-23f3d843febb\" (UID: \"474d2b5a-53c8-43f8-8892-23f3d843febb\") " Nov 01 16:35:50.618693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618630 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-tmp-dir" pod "474d2b5a-53c8-43f8-8892-23f3d843febb" (UID: "474d2b5a-53c8-43f8-8892-23f3d843febb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.618925 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618843 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tmp-dir" pod "aba15e06-9a58-43e7-94c7-8d8aba22d4e5" (UID: "aba15e06-9a58-43e7-94c7-8d8aba22d4e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.618925 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618917 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.619058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618939 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.619058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618956 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.619058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618970 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.619058 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.618985 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.620657 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.620634 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-dshm" pod "aba15e06-9a58-43e7-94c7-8d8aba22d4e5" (UID: "aba15e06-9a58-43e7-94c7-8d8aba22d4e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.620787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.620765 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-dshm" pod "474d2b5a-53c8-43f8-8892-23f3d843febb" (UID: "474d2b5a-53c8-43f8-8892-23f3d843febb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.620867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.620841 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-kube-api-access-tlcsn" pod "aba15e06-9a58-43e7-94c7-8d8aba22d4e5" (UID: "aba15e06-9a58-43e7-94c7-8d8aba22d4e5"). InnerVolumeSpecName "kube-api-access-tlcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:50.620920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.620875 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tls-certs" pod "aba15e06-9a58-43e7-94c7-8d8aba22d4e5" (UID: "aba15e06-9a58-43e7-94c7-8d8aba22d4e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:50.621274 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.621256 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474d2b5a-53c8-43f8-8892-23f3d843febb-tls-certs" pod "474d2b5a-53c8-43f8-8892-23f3d843febb" (UID: "474d2b5a-53c8-43f8-8892-23f3d843febb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:50.621573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.621552 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474d2b5a-53c8-43f8-8892-23f3d843febb-kube-api-access-422qm" pod "474d2b5a-53c8-43f8-8892-23f3d843febb" (UID: "474d2b5a-53c8-43f8-8892-23f3d843febb"). InnerVolumeSpecName "kube-api-access-422qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:50.720212 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720172 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624d676d-43a4-4cd8-8200-33a705076290-kube-api-access-7z9lr\" (UniqueName: \"kubernetes.io/projected/624d676d-43a4-4cd8-8200-33a705076290-kube-api-access-7z9lr\") pod \"624d676d-43a4-4cd8-8200-33a705076290\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " Nov 01 16:35:50.720417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720245 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-dshm\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-dshm\") pod \"624d676d-43a4-4cd8-8200-33a705076290\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " Nov 01 16:35:50.720417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720311 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-model-cache\") pod \"624d676d-43a4-4cd8-8200-33a705076290\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " Nov 01 16:35:50.720417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720351 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-tmp-dir\") pod \"624d676d-43a4-4cd8-8200-33a705076290\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " Nov 01 16:35:50.720417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720382 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/624d676d-43a4-4cd8-8200-33a705076290-tls-certs\" (UniqueName: \"kubernetes.io/secret/624d676d-43a4-4cd8-8200-33a705076290-tls-certs\") pod \"624d676d-43a4-4cd8-8200-33a705076290\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " Nov 01 16:35:50.720645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720423 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-home\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-home\") pod \"624d676d-43a4-4cd8-8200-33a705076290\" (UID: \"624d676d-43a4-4cd8-8200-33a705076290\") " Nov 01 16:35:50.720645 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720587 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-model-cache" pod "624d676d-43a4-4cd8-8200-33a705076290" (UID: "624d676d-43a4-4cd8-8200-33a705076290"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.720761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720659 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-tmp-dir" pod "624d676d-43a4-4cd8-8200-33a705076290" (UID: "624d676d-43a4-4cd8-8200-33a705076290"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.720761 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.720748 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-home" pod "624d676d-43a4-4cd8-8200-33a705076290" (UID: "624d676d-43a4-4cd8-8200-33a705076290"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.721116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721081 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721118 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlcsn\" (UniqueName: \"kubernetes.io/projected/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-kube-api-access-tlcsn\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721133 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721155 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d2b5a-53c8-43f8-8892-23f3d843febb-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721168 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d2b5a-53c8-43f8-8892-23f3d843febb-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721180 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba15e06-9a58-43e7-94c7-8d8aba22d4e5-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721194 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-422qm\" (UniqueName: \"kubernetes.io/projected/474d2b5a-53c8-43f8-8892-23f3d843febb-kube-api-access-422qm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721207 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721220 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.721240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.721231 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.722537 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.722512 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-dshm" pod "624d676d-43a4-4cd8-8200-33a705076290" (UID: "624d676d-43a4-4cd8-8200-33a705076290"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:50.722830 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.722811 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624d676d-43a4-4cd8-8200-33a705076290-tls-certs" pod "624d676d-43a4-4cd8-8200-33a705076290" (UID: "624d676d-43a4-4cd8-8200-33a705076290"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:50.722900 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.722829 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624d676d-43a4-4cd8-8200-33a705076290-kube-api-access-7z9lr" pod "624d676d-43a4-4cd8-8200-33a705076290" (UID: "624d676d-43a4-4cd8-8200-33a705076290"). InnerVolumeSpecName "kube-api-access-7z9lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:50.822511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.822430 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z9lr\" (UniqueName: \"kubernetes.io/projected/624d676d-43a4-4cd8-8200-33a705076290-kube-api-access-7z9lr\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.822511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.822458 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624d676d-43a4-4cd8-8200-33a705076290-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:50.822511 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:50.822468 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624d676d-43a4-4cd8-8200-33a705076290-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:51.300054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.300008 4911 generic.go:356] "Generic (PLEG): container finished" podID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerID="cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c" exitCode=0 Nov 01 16:35:51.300477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.300067 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" Nov 01 16:35:51.300477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.300094 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" event={"ID":"474d2b5a-53c8-43f8-8892-23f3d843febb","Type":"ContainerDied","Data":"cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c"} Nov 01 16:35:51.300477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.300133 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8" event={"ID":"474d2b5a-53c8-43f8-8892-23f3d843febb","Type":"ContainerDied","Data":"47b77d27900b0c2fc28cd5b70c54d303bb78c37cc9c69dbe2edf900bac64e1ea"} Nov 01 16:35:51.300477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.300150 4911 scope.go:122] "RemoveContainer" containerID="cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c" Nov 01 16:35:51.301587 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.301568 4911 generic.go:356] "Generic (PLEG): container finished" podID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerID="01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad" exitCode=0 Nov 01 16:35:51.301675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.301624 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" Nov 01 16:35:51.301729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.301653 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" event={"ID":"aba15e06-9a58-43e7-94c7-8d8aba22d4e5","Type":"ContainerDied","Data":"01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad"} Nov 01 16:35:51.301729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.301704 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97" event={"ID":"aba15e06-9a58-43e7-94c7-8d8aba22d4e5","Type":"ContainerDied","Data":"f0c9be7d7ed498b3a68205a4bb67c471375b62f803f7db4e55a3ff3c360fb23b"} Nov 01 16:35:51.302946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.302922 4911 generic.go:356] "Generic (PLEG): container finished" podID="624d676d-43a4-4cd8-8200-33a705076290" containerID="3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617" exitCode=0 Nov 01 16:35:51.303099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.302974 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" Nov 01 16:35:51.303099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.302985 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" event={"ID":"624d676d-43a4-4cd8-8200-33a705076290","Type":"ContainerDied","Data":"3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617"} Nov 01 16:35:51.303099 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.303008 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg" event={"ID":"624d676d-43a4-4cd8-8200-33a705076290","Type":"ContainerDied","Data":"061626ccdacb87f924437e98ca3251311a156628ef61572c8779296d2e42ab16"} Nov 01 16:35:51.312978 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.312945 4911 scope.go:122] "RemoveContainer" containerID="cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c" Nov 01 16:35:51.313293 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:51.313273 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c\": container with ID starting with cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c not found: ID does not exist" containerID="cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c" Nov 01 16:35:51.313366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.313301 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c"} err="failed to get container status \"cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c\": rpc error: code = NotFound desc = could not find container \"cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c\": container with ID starting with cd00ac4d24ef9b4ae1ab7f5bed12df9477f1f4d1a3eefece8c15e2072259570c not found: ID does not exist" Nov 01 16:35:51.313366 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.313322 4911 scope.go:122] "RemoveContainer" containerID="01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad" Nov 01 16:35:51.322422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.322404 4911 scope.go:122] "RemoveContainer" containerID="01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad" Nov 01 16:35:51.322683 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:51.322659 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad\": container with ID starting with 01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad not found: ID does not exist" containerID="01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad" Nov 01 16:35:51.322750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.322690 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad"} err="failed to get container status \"01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad\": rpc error: code = NotFound desc = could not find container \"01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad\": container with ID starting with 01f42acb204b12d78bdb3eae7f0e4ac11f8d56e3ff1cbd108e6b741f7af8c3ad not found: ID does not exist" Nov 01 16:35:51.322750 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.322705 4911 scope.go:122] "RemoveContainer" containerID="3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617" Nov 01 16:35:51.331958 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.331934 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8"] Nov 01 16:35:51.332543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.332526 4911 scope.go:122] "RemoveContainer" containerID="3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617" Nov 01 16:35:51.332804 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:35:51.332782 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617\": container with ID starting with 3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617 not found: ID does not exist" containerID="3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617" Nov 01 16:35:51.332867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.332811 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617"} err="failed to get container status \"3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617\": rpc error: code = NotFound desc = could not find container \"3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617\": container with ID starting with 3108574103b64964fa6d5d7e9fb81ddcdde661f547427d3ec19a076cbb872617 not found: ID does not exist" Nov 01 16:35:51.336013 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.335993 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/simulator-w5-164db6-kserve-696555ff9c-wnsg8"] Nov 01 16:35:51.346859 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.346831 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97"] Nov 01 16:35:51.350768 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.350745 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-2-w5-164db6-kserve-64c769844d-cnz97"] Nov 01 16:35:51.361646 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.361623 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg"] Nov 01 16:35:51.365452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.365430 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-164db6/distinct-w5-164db6-kserve-d94795f-zjdkg"] Nov 01 16:35:51.964960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.964920 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" path="/var/lib/kubelet/pods/474d2b5a-53c8-43f8-8892-23f3d843febb/volumes" Nov 01 16:35:51.965407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.965390 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624d676d-43a4-4cd8-8200-33a705076290" path="/var/lib/kubelet/pods/624d676d-43a4-4cd8-8200-33a705076290/volumes" Nov 01 16:35:51.965767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:51.965755 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" path="/var/lib/kubelet/pods/aba15e06-9a58-43e7-94c7-8d8aba22d4e5/volumes" Nov 01 16:35:53.450787 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:35:53.450749 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" podUID="3475b4c0-cc38-47d7-b344-8c2170eee46e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:36:03.027399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:03.027358 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b"] Nov 01 16:36:03.028308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:03.028253 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" containerID="cri-o://6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190" gracePeriod=30 Nov 01 16:36:03.451249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:03.451150 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" podUID="3475b4c0-cc38-47d7-b344-8c2170eee46e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:36:05.473671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.473632 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-646677ff99-h225q"] Nov 01 16:36:05.474126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.473840 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-646677ff99-h225q" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerName="manager" containerID="cri-o://31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef" gracePeriod=30 Nov 01 16:36:05.745899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.745868 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 16:36:05.767899 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.767866 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 16:36:05.863776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.863739 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3475b4c0-cc38-47d7-b344-8c2170eee46e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475b4c0-cc38-47d7-b344-8c2170eee46e-plugins-config-volume\") pod \"3475b4c0-cc38-47d7-b344-8c2170eee46e\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " Nov 01 16:36:05.863776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.863778 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3475b4c0-cc38-47d7-b344-8c2170eee46e-kube-api-access-t8wl5\" (UniqueName: \"kubernetes.io/projected/3475b4c0-cc38-47d7-b344-8c2170eee46e-kube-api-access-t8wl5\") pod \"3475b4c0-cc38-47d7-b344-8c2170eee46e\" (UID: \"3475b4c0-cc38-47d7-b344-8c2170eee46e\") " Nov 01 16:36:05.863999 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.863898 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-kube-api-access-7vnzv\" (UniqueName: \"kubernetes.io/projected/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-kube-api-access-7vnzv\") pod \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " Nov 01 16:36:05.864072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.864003 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-metrics-certs\") pod \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " Nov 01 16:36:05.864143 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.864033 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-webhook-cert\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-webhook-cert\") pod \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\" (UID: \"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755\") " Nov 01 16:36:05.864242 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.864214 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3475b4c0-cc38-47d7-b344-8c2170eee46e-plugins-config-volume" pod "3475b4c0-cc38-47d7-b344-8c2170eee46e" (UID: "3475b4c0-cc38-47d7-b344-8c2170eee46e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:36:05.864459 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.864431 4911 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3475b4c0-cc38-47d7-b344-8c2170eee46e-plugins-config-volume\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:05.866025 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.865993 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3475b4c0-cc38-47d7-b344-8c2170eee46e-kube-api-access-t8wl5" pod "3475b4c0-cc38-47d7-b344-8c2170eee46e" (UID: "3475b4c0-cc38-47d7-b344-8c2170eee46e"). InnerVolumeSpecName "kube-api-access-t8wl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:05.866166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.866025 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-kube-api-access-7vnzv" pod "df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" (UID: "df2ea6f2-d2a7-41eb-8bd4-1993ecc75755"). InnerVolumeSpecName "kube-api-access-7vnzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:05.866166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.866056 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-metrics-certs" pod "df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" (UID: "df2ea6f2-d2a7-41eb-8bd4-1993ecc75755"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:05.866166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.866141 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-webhook-cert" pod "df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" (UID: "df2ea6f2-d2a7-41eb-8bd4-1993ecc75755"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:05.964890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.964852 4911 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-metrics-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:05.964890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.964883 4911 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-webhook-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:05.965131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.964896 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wl5\" (UniqueName: \"kubernetes.io/projected/3475b4c0-cc38-47d7-b344-8c2170eee46e-kube-api-access-t8wl5\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:05.965131 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:05.964925 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnzv\" (UniqueName: \"kubernetes.io/projected/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755-kube-api-access-7vnzv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:06.387973 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.387939 4911 generic.go:356] "Generic (PLEG): container finished" podID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerID="31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef" exitCode=0 Nov 01 16:36:06.388182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.387987 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-646677ff99-h225q" Nov 01 16:36:06.388182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.388031 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-h225q" event={"ID":"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755","Type":"ContainerDied","Data":"31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef"} Nov 01 16:36:06.388182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.388118 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-646677ff99-h225q" event={"ID":"df2ea6f2-d2a7-41eb-8bd4-1993ecc75755","Type":"ContainerDied","Data":"43e374c31fae90bc4480ec2f79d54a291dbf505a71f7f481ef744e8259d77fc3"} Nov 01 16:36:06.388182 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.388152 4911 scope.go:122] "RemoveContainer" containerID="31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef" Nov 01 16:36:06.389540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.389515 4911 generic.go:356] "Generic (PLEG): container finished" podID="3475b4c0-cc38-47d7-b344-8c2170eee46e" containerID="188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589" exitCode=0 Nov 01 16:36:06.389692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.389555 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" Nov 01 16:36:06.389692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.389611 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" event={"ID":"3475b4c0-cc38-47d7-b344-8c2170eee46e","Type":"ContainerDied","Data":"188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589"} Nov 01 16:36:06.389692 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.389665 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6" event={"ID":"3475b4c0-cc38-47d7-b344-8c2170eee46e","Type":"ContainerDied","Data":"5d548131d6eafe0106431bf7927ca9c5d07019507be175fec0ea7ed1e47a0da8"} Nov 01 16:36:06.399458 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.399440 4911 scope.go:122] "RemoveContainer" containerID="34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8" Nov 01 16:36:06.408124 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.408106 4911 scope.go:122] "RemoveContainer" containerID="31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef" Nov 01 16:36:06.408405 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:36:06.408384 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef\": container with ID starting with 31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef not found: ID does not exist" containerID="31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef" Nov 01 16:36:06.408498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.408415 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef"} err="failed to get container status \"31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef\": rpc error: code = NotFound desc = could not find container \"31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef\": container with ID starting with 31a15899b3a1cd76fd2bc85655fc8465f4364a16e8809341623398b92ed13eef not found: ID does not exist" Nov 01 16:36:06.408498 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.408437 4911 scope.go:122] "RemoveContainer" containerID="34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8" Nov 01 16:36:06.408687 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:36:06.408666 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8\": container with ID starting with 34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8 not found: ID does not exist" containerID="34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8" Nov 01 16:36:06.408731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.408697 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8"} err="failed to get container status \"34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8\": rpc error: code = NotFound desc = could not find container \"34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8\": container with ID starting with 34e881f83b2f9ab9d4e117cc453991a80998960db6d048ae59f223c11a2508c8 not found: ID does not exist" Nov 01 16:36:06.408731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.408717 4911 scope.go:122] "RemoveContainer" containerID="188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589" Nov 01 16:36:06.412470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.412445 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-646677ff99-h225q"] Nov 01 16:36:06.416966 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.416945 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-646677ff99-h225q"] Nov 01 16:36:06.419305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.419288 4911 scope.go:122] "RemoveContainer" containerID="188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589" Nov 01 16:36:06.419554 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:36:06.419529 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589\": container with ID starting with 188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589 not found: ID does not exist" containerID="188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589" Nov 01 16:36:06.419607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.419563 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589"} err="failed to get container status \"188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589\": rpc error: code = NotFound desc = could not find container \"188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589\": container with ID starting with 188fa98b588620529ed6da763c0e1cd59338cf2904528b0bac288e1e312a7589 not found: ID does not exist" Nov 01 16:36:06.426826 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.426797 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6"] Nov 01 16:36:06.431243 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:06.431222 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-164db6-649d7fd8f5-2ttc6"] Nov 01 16:36:07.968281 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:07.968241 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3475b4c0-cc38-47d7-b344-8c2170eee46e" path="/var/lib/kubelet/pods/3475b4c0-cc38-47d7-b344-8c2170eee46e/volumes" Nov 01 16:36:07.968852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:07.968827 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" path="/var/lib/kubelet/pods/df2ea6f2-d2a7-41eb-8bd4-1993ecc75755/volumes" Nov 01 16:36:08.320712 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.320686 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:36:08.420701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.420666 4911 generic.go:356] "Generic (PLEG): container finished" podID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerID="6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190" exitCode=0 Nov 01 16:36:08.420849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.420711 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" Nov 01 16:36:08.420849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.420747 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" event={"ID":"b57cb508-00c4-4520-a10e-f84d608df2ce","Type":"ContainerDied","Data":"6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190"} Nov 01 16:36:08.420849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.420801 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b" event={"ID":"b57cb508-00c4-4520-a10e-f84d608df2ce","Type":"ContainerDied","Data":"6d2a10652eaaef1ed54678ec20622b1b4b0991af7be91f00dea2f712c3f73337"} Nov 01 16:36:08.420849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.420824 4911 scope.go:122] "RemoveContainer" containerID="6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190" Nov 01 16:36:08.431434 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.431405 4911 scope.go:122] "RemoveContainer" containerID="6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190" Nov 01 16:36:08.431695 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:36:08.431672 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190\": container with ID starting with 6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190 not found: ID does not exist" containerID="6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190" Nov 01 16:36:08.431745 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.431705 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190"} err="failed to get container status \"6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190\": rpc error: code = NotFound desc = could not find container \"6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190\": container with ID starting with 6a0c020e6cf133e9d364b60c6d33519ddc48b20b27041867795c34149aed5190 not found: ID does not exist" Nov 01 16:36:08.492444 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492399 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-envoy\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492493 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-credential-socket\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492526 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-certs\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492567 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-podinfo\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492636 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492607 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-socket\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492645 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-data\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492679 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57cb508-00c4-4520-a10e-f84d608df2ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57cb508-00c4-4520-a10e-f84d608df2ce-istiod-ca-cert\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492744 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-token\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-token\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.492870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492826 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-kube-api-access-vjdcc\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-kube-api-access-vjdcc\") pod \"b57cb508-00c4-4520-a10e-f84d608df2ce\" (UID: \"b57cb508-00c4-4520-a10e-f84d608df2ce\") " Nov 01 16:36:08.493142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492868 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-credential-socket" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.493142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.492887 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-certs" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.493142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.493023 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-socket" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.493304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.493220 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.493304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.493244 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.493304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.493260 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.493304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.493280 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57cb508-00c4-4520-a10e-f84d608df2ce-istiod-ca-cert" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:36:08.493559 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.493535 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-data" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.495542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.495496 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-envoy" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.495798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.495766 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-podinfo" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:36:08.495980 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.495949 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-kube-api-access-vjdcc" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "kube-api-access-vjdcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:08.496143 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.495958 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-token" pod "b57cb508-00c4-4520-a10e-f84d608df2ce" (UID: "b57cb508-00c4-4520-a10e-f84d608df2ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:08.593959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.593872 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.593959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.593908 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.593959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.593918 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.593959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.593928 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b57cb508-00c4-4520-a10e-f84d608df2ce-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.593959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.593940 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.593959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.593949 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjdcc\" (UniqueName: \"kubernetes.io/projected/b57cb508-00c4-4520-a10e-f84d608df2ce-kube-api-access-vjdcc\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.744647 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.744617 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b"] Nov 01 16:36:08.749982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:08.749944 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kw58b"] Nov 01 16:36:09.964604 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:09.964571 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" path="/var/lib/kubelet/pods/b57cb508-00c4-4520-a10e-f84d608df2ce/volumes" Nov 01 16:36:10.865449 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:36:10.865415 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:36:35.185984 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.185900 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm"] Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186296 4911 cpu_manager.go:414] "containerMap: removing container" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186310 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186349 4911 cpu_manager.go:414] "containerMap: removing container" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerName="manager" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186357 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerName="manager" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186376 4911 cpu_manager.go:414] "containerMap: removing container" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186386 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186394 4911 cpu_manager.go:414] "containerMap: removing container" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerName="manager" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186399 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerName="manager" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186407 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186411 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186420 4911 cpu_manager.go:414] "containerMap: removing container" podUID="3475b4c0-cc38-47d7-b344-8c2170eee46e" containerName="payload-pre-processing" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186426 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3475b4c0-cc38-47d7-b344-8c2170eee46e" containerName="payload-pre-processing" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186435 4911 cpu_manager.go:414] "containerMap: removing container" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186439 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186449 4911 cpu_manager.go:414] "containerMap: removing container" podUID="af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" containerName="revoke-keys" Nov 01 16:36:35.186462 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186453 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" containerName="revoke-keys" Nov 01 16:36:35.186994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186514 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerName="manager" Nov 01 16:36:35.186994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186522 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba15e06-9a58-43e7-94c7-8d8aba22d4e5" containerName="main" Nov 01 16:36:35.186994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186530 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57cb508-00c4-4520-a10e-f84d608df2ce" containerName="istio-proxy" Nov 01 16:36:35.186994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186536 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="3475b4c0-cc38-47d7-b344-8c2170eee46e" containerName="payload-pre-processing" Nov 01 16:36:35.186994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186544 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="474d2b5a-53c8-43f8-8892-23f3d843febb" containerName="main" Nov 01 16:36:35.186994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186549 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" containerName="revoke-keys" Nov 01 16:36:35.186994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.186556 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="624d676d-43a4-4cd8-8200-33a705076290" containerName="main" Nov 01 16:36:35.190851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.190828 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.203149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.203119 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm"] Nov 01 16:36:35.290655 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290615 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf78h\" (UniqueName: \"kubernetes.io/projected/a0181ade-77fb-4c16-adda-22dca4ce2524-kube-api-access-mf78h\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.290851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290677 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.290851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290753 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0181ade-77fb-4c16-adda-22dca4ce2524-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.290851 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290793 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.290985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290875 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.290985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290895 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.290985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290920 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.290985 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290963 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.291160 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.290997 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.391579 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391531 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf78h\" (UniqueName: \"kubernetes.io/projected/a0181ade-77fb-4c16-adda-22dca4ce2524-kube-api-access-mf78h\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.391743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391642 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.391743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391665 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0181ade-77fb-4c16-adda-22dca4ce2524-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.391743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391683 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.391743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391712 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.391743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391740 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391782 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391806 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.391834 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392288 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.392259 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.392304 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392370 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.392337 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392480 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.392417 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.392545 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.392526 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0181ade-77fb-4c16-adda-22dca4ce2524-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.394381 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.394359 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.394562 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.394547 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.399429 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.399407 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0181ade-77fb-4c16-adda-22dca4ce2524-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.399630 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.399611 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf78h\" (UniqueName: \"kubernetes.io/projected/a0181ade-77fb-4c16-adda-22dca4ce2524-kube-api-access-mf78h\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vhjzm\" (UID: \"a0181ade-77fb-4c16-adda-22dca4ce2524\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.505031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.504997 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:35.785822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.785792 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm"] Nov 01 16:36:35.787128 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:36:35.787098 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0181ade_77fb_4c16_adda_22dca4ce2524.slice/crio-5cd2c4b0e6907155cfe55b5cda4e3652402df9bed49f7cbad6518679b47fd4c9 WatchSource:0}: Error finding container 5cd2c4b0e6907155cfe55b5cda4e3652402df9bed49f7cbad6518679b47fd4c9: Status 404 returned error can't find the container with id 5cd2c4b0e6907155cfe55b5cda4e3652402df9bed49f7cbad6518679b47fd4c9 Nov 01 16:36:35.789304 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.789269 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:36:35.789427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.789333 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:36:35.789427 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:35.789362 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:36:36.580754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:36.580719 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" event={"ID":"a0181ade-77fb-4c16-adda-22dca4ce2524","Type":"ContainerStarted","Data":"61dd88b6b2bc610eaf4f9e0f8d412f2e278d82c09eb6b1cbc1a78189d2fe918d"} Nov 01 16:36:36.581229 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:36.580774 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" event={"ID":"a0181ade-77fb-4c16-adda-22dca4ce2524","Type":"ContainerStarted","Data":"5cd2c4b0e6907155cfe55b5cda4e3652402df9bed49f7cbad6518679b47fd4c9"} Nov 01 16:36:36.603202 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:36.603152 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podStartSLOduration=1.6031395929999999 podStartE2EDuration="1.603139593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:36:36.600353014 +0000 UTC m=+2936.744807656" watchObservedRunningTime="2026-11-01 16:36:36.603139593 +0000 UTC m=+2936.747594233" Nov 01 16:36:37.505259 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:37.505218 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:37.506920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:37.506892 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vhjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 16:36:37.507097 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:37.506942 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podUID="a0181ade-77fb-4c16-adda-22dca4ce2524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 16:36:38.505952 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:38.505903 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vhjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 16:36:38.506503 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:38.505969 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podUID="a0181ade-77fb-4c16-adda-22dca4ce2524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 16:36:39.506478 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:39.506438 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vhjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 16:36:39.506890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:39.506510 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podUID="a0181ade-77fb-4c16-adda-22dca4ce2524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 16:36:40.506247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:40.506207 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vhjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 16:36:40.506533 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:40.506276 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podUID="a0181ade-77fb-4c16-adda-22dca4ce2524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 16:36:40.887426 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:36:40.887331 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:36:41.505363 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:41.505319 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vhjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 16:36:41.505583 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:41.505418 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podUID="a0181ade-77fb-4c16-adda-22dca4ce2524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 16:36:42.505757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:42.505719 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vhjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 01 16:36:42.506210 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:42.505778 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podUID="a0181ade-77fb-4c16-adda-22dca4ce2524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 01 16:36:42.747136 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:42.747101 4911 scope.go:122] "RemoveContainer" containerID="d954f47e6316cf7881da6e0a5ceddd9f9bcb3982de7ce2cd6908fb62cbddbdcc" Nov 01 16:36:42.756323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:42.756299 4911 scope.go:122] "RemoveContainer" containerID="eb861834938f451641faf714d9ae2aff5e8b00cddf395aa22ec34b0753bbcedd" Nov 01 16:36:43.982657 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:43.982618 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:43.983168 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:43.983075 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:36:43.983942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:36:43.983921 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" Nov 01 16:37:10.866966 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:37:10.866918 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:37:40.799841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:37:40.799811 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:37:40.805409 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:37:40.805384 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:37:40.865620 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:37:40.865581 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:38:10.866394 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:38:10.866354 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:38:15.024288 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:15.024245 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vhjzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:38:15.024819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:15.024308 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vhjzm" podUID="a0181ade-77fb-4c16-adda-22dca4ce2524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:38:35.331454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.331413 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq"] Nov 01 16:38:35.331965 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.331938 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2ea6f2-d2a7-41eb-8bd4-1993ecc75755" containerName="manager" Nov 01 16:38:35.335236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.335218 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.353704 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.353673 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq"] Nov 01 16:38:35.428693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.428651 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.428693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.428693 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.428962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.428723 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db534cd4-9441-4dab-abe4-f8109f12f529-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.428962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.428786 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.428962 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.428864 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sznk6\" (UniqueName: \"kubernetes.io/projected/db534cd4-9441-4dab-abe4-f8109f12f529-kube-api-access-sznk6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.429228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.428955 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db534cd4-9441-4dab-abe4-f8109f12f529-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.429228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.429028 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db534cd4-9441-4dab-abe4-f8109f12f529-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.429228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.429105 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.429228 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.429147 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.530669 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530623 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.530875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530680 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.530875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530719 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.530875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530748 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.530875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530786 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db534cd4-9441-4dab-abe4-f8109f12f529-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.530875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530812 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.530875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530872 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznk6\" (UniqueName: \"kubernetes.io/projected/db534cd4-9441-4dab-abe4-f8109f12f529-kube-api-access-sznk6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.531277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530921 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db534cd4-9441-4dab-abe4-f8109f12f529-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.531277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.530970 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db534cd4-9441-4dab-abe4-f8109f12f529-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.531277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.531136 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.531277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.531164 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.531277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.531244 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.531578 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.531305 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.531784 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.531763 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db534cd4-9441-4dab-abe4-f8109f12f529-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.533191 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.533160 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db534cd4-9441-4dab-abe4-f8109f12f529-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.533497 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.533479 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db534cd4-9441-4dab-abe4-f8109f12f529-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.539767 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.539746 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db534cd4-9441-4dab-abe4-f8109f12f529-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.540251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.540235 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznk6\" (UniqueName: \"kubernetes.io/projected/db534cd4-9441-4dab-abe4-f8109f12f529-kube-api-access-sznk6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-s62zq\" (UID: \"db534cd4-9441-4dab-abe4-f8109f12f529\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.648404 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.648320 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:35.934385 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.934351 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq"] Nov 01 16:38:35.936322 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:38:35.936287 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb534cd4_9441_4dab_abe4_f8109f12f529.slice/crio-a2fef08c7676158cf69b5f29f57552f8065848e123393ac0bbabacd898ab2410 WatchSource:0}: Error finding container a2fef08c7676158cf69b5f29f57552f8065848e123393ac0bbabacd898ab2410: Status 404 returned error can't find the container with id a2fef08c7676158cf69b5f29f57552f8065848e123393ac0bbabacd898ab2410 Nov 01 16:38:35.938856 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.938824 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.938942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.938890 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.938942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:35.938920 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:36.223708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:36.223618 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" event={"ID":"db534cd4-9441-4dab-abe4-f8109f12f529","Type":"ContainerStarted","Data":"bb99b5e38b04409b4feed354f06038b3ed1fb65b462ecaafec2b3c21fa78a287"} Nov 01 16:38:36.223708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:36.223691 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" event={"ID":"db534cd4-9441-4dab-abe4-f8109f12f529","Type":"ContainerStarted","Data":"a2fef08c7676158cf69b5f29f57552f8065848e123393ac0bbabacd898ab2410"} Nov 01 16:38:36.248737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:36.248656 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podStartSLOduration=1.248636446 podStartE2EDuration="1.248636446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:38:36.244367879 +0000 UTC m=+3056.388822522" watchObservedRunningTime="2026-11-01 16:38:36.248636446 +0000 UTC m=+3056.393091088" Nov 01 16:38:36.649106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:36.649061 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:36.660065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:36.651140 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s62zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 16:38:36.660065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:36.651238 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podUID="db534cd4-9441-4dab-abe4-f8109f12f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 16:38:37.648852 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:37.648807 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s62zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 16:38:37.649093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:37.648886 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podUID="db534cd4-9441-4dab-abe4-f8109f12f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 16:38:38.662560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:38.662516 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s62zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 16:38:38.663215 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:38.663176 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podUID="db534cd4-9441-4dab-abe4-f8109f12f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 16:38:39.672227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:39.672181 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s62zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 16:38:39.672860 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:39.672821 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podUID="db534cd4-9441-4dab-abe4-f8109f12f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 16:38:40.649032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:40.648993 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s62zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 16:38:40.649292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:40.649089 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podUID="db534cd4-9441-4dab-abe4-f8109f12f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 16:38:40.886658 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:38:40.886616 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:38:41.671063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:41.670509 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s62zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 16:38:41.671063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:41.670575 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podUID="db534cd4-9441-4dab-abe4-f8109f12f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 16:38:42.648804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:42.648763 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-s62zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 01 16:38:42.649364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:42.648817 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" podUID="db534cd4-9441-4dab-abe4-f8109f12f529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 01 16:38:44.106630 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:44.106595 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:44.107179 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:44.107071 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:38:44.107781 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:38:44.107759 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-s62zq" Nov 01 16:39:00.946510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:00.946464 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x"] Nov 01 16:39:00.947002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:00.946676 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" podUID="b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" containerName="discovery" containerID="cri-o://349ee28e395a595dd49418a4b0e0ac47ad424995f1700bd2b4f748d44660539e" gracePeriod=30 Nov 01 16:39:00.970460 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:00.969130 4911 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2qs8x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:8080/ready\": dial tcp 10.132.0.110:8080: connect: connection refused" start-of-body= Nov 01 16:39:00.970460 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:00.969185 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" podUID="b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.110:8080/ready\": dial tcp 10.132.0.110:8080: connect: connection refused" Nov 01 16:39:01.434564 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.434509 4911 generic.go:356] "Generic (PLEG): container finished" podID="b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" containerID="349ee28e395a595dd49418a4b0e0ac47ad424995f1700bd2b4f748d44660539e" exitCode=0 Nov 01 16:39:01.434971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.434944 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" event={"ID":"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5","Type":"ContainerDied","Data":"349ee28e395a595dd49418a4b0e0ac47ad424995f1700bd2b4f748d44660539e"} Nov 01 16:39:01.499064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.496736 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:39:01.537555 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.537455 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-cacerts\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-cacerts\") pod \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " Nov 01 16:39:01.537555 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.537505 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-ca-configmap\") pod \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " Nov 01 16:39:01.537802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.537555 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-kubeconfig\") pod \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " Nov 01 16:39:01.537802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.537590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-dns-cert\") pod \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " Nov 01 16:39:01.537802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.537638 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-local-certs\") pod \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " Nov 01 16:39:01.537802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.537666 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-token\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-token\") pod \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " Nov 01 16:39:01.537802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.537726 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-kube-api-access-v9g57\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-kube-api-access-v9g57\") pod \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\" (UID: \"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5\") " Nov 01 16:39:01.539269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.539213 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-ca-configmap" pod "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" (UID: "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:39:01.540981 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.540909 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-kubeconfig" pod "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" (UID: "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:39:01.541183 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.540951 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-kube-api-access-v9g57" pod "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" (UID: "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5"). InnerVolumeSpecName "kube-api-access-v9g57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:39:01.541506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.541443 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-local-certs" pod "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" (UID: "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:39:01.541804 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.541701 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-dns-cert" pod "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" (UID: "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:39:01.541955 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.541750 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-token" pod "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" (UID: "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:39:01.542106 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.541952 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-cacerts" pod "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" (UID: "b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:39:01.639528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.639488 4911 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-dns-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:01.639528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.639527 4911 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-local-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:01.639775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.639540 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:01.639775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.639568 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9g57\" (UniqueName: \"kubernetes.io/projected/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-kube-api-access-v9g57\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:01.639775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.639582 4911 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-cacerts\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:01.639775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.639594 4911 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-csr-ca-configmap\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:01.639775 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:01.639608 4911 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5-istio-kubeconfig\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:02.442674 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:02.442592 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" Nov 01 16:39:02.443138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:02.442595 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x" event={"ID":"b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5","Type":"ContainerDied","Data":"1235db931af85cd067e8140d4f7124b4adb336219ee2220ea05ea2b1434cc4e1"} Nov 01 16:39:02.443138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:02.442728 4911 scope.go:122] "RemoveContainer" containerID="349ee28e395a595dd49418a4b0e0ac47ad424995f1700bd2b4f748d44660539e" Nov 01 16:39:02.464189 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:02.464157 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x"] Nov 01 16:39:02.469360 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:02.469331 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2qs8x"] Nov 01 16:39:03.023738 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:03.023697 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r"] Nov 01 16:39:03.025802 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:03.025780 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77d75604-9nq8r"] Nov 01 16:39:03.964586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:03.964553 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" path="/var/lib/kubelet/pods/b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5/volumes" Nov 01 16:39:03.965024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:03.965007 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299" path="/var/lib/kubelet/pods/c0f9ee1e-83a0-4b6b-acb8-84d6ecf0b299/volumes" Nov 01 16:39:10.865910 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:39:10.865871 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:39:40.867308 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:39:40.867265 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:39:59.283065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:59.263245 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9rb5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:39:59.283065 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:39:59.263335 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podUID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:40:10.865839 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:40:10.865800 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:40:40.866406 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:40:40.866364 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:40:42.855658 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:40:42.855622 4911 scope.go:122] "RemoveContainer" containerID="f76ba69347b601c3891ab3e6060a718706e5a98a68144b67ee8f6fe9db1f09bc" Nov 01 16:40:42.864790 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:40:42.864766 4911 scope.go:122] "RemoveContainer" containerID="d0f0f80afa6172297259155031ede4b9ad2ded64cdba2e3d5f9af802996a99c4" Nov 01 16:40:45.026135 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:40:45.026095 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq"] Nov 01 16:40:45.028599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:40:45.028571 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd4960a2-pfgsq"] Nov 01 16:40:45.964918 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:40:45.964883 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af98d2fb-1d3e-4c66-8644-f80bc5e0a16b" path="/var/lib/kubelet/pods/af98d2fb-1d3e-4c66-8644-f80bc5e0a16b/volumes" Nov 01 16:41:10.868460 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:41:10.868424 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:41:40.865882 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:41:40.865841 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:41:42.888282 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:41:42.888241 4911 scope.go:122] "RemoveContainer" containerID="1443e2bcb41fee671febc0e3952e8a56bcadd71acd32938489632b41988e3af1" Nov 01 16:42:10.865104 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:42:10.865066 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:42:14.270064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:14.269153 4911 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-9rb5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:42:14.270064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:14.269228 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-9rb5l" podUID="f37e3bdc-1a02-41cb-8c76-61d11cbd86b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:42:20.407186 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.407143 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5"] Nov 01 16:42:20.407800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.407773 4911 cpu_manager.go:414] "containerMap: removing container" podUID="b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" containerName="discovery" Nov 01 16:42:20.407800 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.407795 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" containerName="discovery" Nov 01 16:42:20.408128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.407934 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61ca4b5-2b3b-4017-adfb-a6f70e1c4de5" containerName="discovery" Nov 01 16:42:20.411199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.411173 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.423003 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.422977 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ee2b3d-openshift-default-dockercfg-fhg9n\"" Nov 01 16:42:20.428114 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.428084 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5"] Nov 01 16:42:20.465374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465336 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-data\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465374 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465381 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-socket\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465476 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxsch\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-kube-api-access-gxsch\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465494 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-istio-token\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465513 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-certs\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465529 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-envoy\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465650 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-credential-socket\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465679 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f744fccd-1edc-42c6-bf53-2973623e1179-istio-podinfo\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.465723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.465696 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f744fccd-1edc-42c6-bf53-2973623e1179-istiod-ca-cert\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.578540 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.578494 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-credential-socket\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.578822 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.578800 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f744fccd-1edc-42c6-bf53-2973623e1179-istio-podinfo\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.578926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.578913 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f744fccd-1edc-42c6-bf53-2973623e1179-istiod-ca-cert\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.579081 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.579067 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-data\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.579203 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.579190 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-socket\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.579432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.579404 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxsch\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-kube-api-access-gxsch\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.579584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.579438 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-istio-token\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.579584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.579467 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-certs\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.579584 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.579494 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-envoy\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.580487 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.580458 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-data\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.581016 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.580916 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-credential-socket\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.585000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.582141 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-socket\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.585000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.582872 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-certs\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.585000 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.583213 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f744fccd-1edc-42c6-bf53-2973623e1179-istiod-ca-cert\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.585671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.585622 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-envoy\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.586591 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.586565 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f744fccd-1edc-42c6-bf53-2973623e1179-istio-podinfo\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.597867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.597826 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxsch\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-kube-api-access-gxsch\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.599433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.599315 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-istio-token\") pod \"e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:20.728866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:20.728775 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:21.008926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.008897 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5"] Nov 01 16:42:21.011715 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:42:21.011690 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf744fccd_1edc_42c6_bf53_2973623e1179.slice/crio-e523f840a11f15eda49ac802c32d43deaccff9c684f23bfc8b005ba789714aab WatchSource:0}: Error finding container e523f840a11f15eda49ac802c32d43deaccff9c684f23bfc8b005ba789714aab: Status 404 returned error can't find the container with id e523f840a11f15eda49ac802c32d43deaccff9c684f23bfc8b005ba789714aab Nov 01 16:42:21.013891 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.013855 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:42:21.013998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.013925 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:42:21.013998 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.013955 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:42:21.642589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.642555 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" event={"ID":"f744fccd-1edc-42c6-bf53-2973623e1179","Type":"ContainerStarted","Data":"9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4"} Nov 01 16:42:21.643029 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.642607 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" event={"ID":"f744fccd-1edc-42c6-bf53-2973623e1179","Type":"ContainerStarted","Data":"e523f840a11f15eda49ac802c32d43deaccff9c684f23bfc8b005ba789714aab"} Nov 01 16:42:21.664448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.664383 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" podStartSLOduration=1.664363507 podStartE2EDuration="1.664363507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:42:21.66334413 +0000 UTC m=+3281.807798774" watchObservedRunningTime="2026-11-01 16:42:21.664363507 +0000 UTC m=+3281.808818148" Nov 01 16:42:21.729568 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:21.729536 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:22.734557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:22.734528 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:23.652674 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:23.652633 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:23.653674 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:23.653648 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:37.728631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:37.728546 4911 generic.go:356] "Generic (PLEG): container finished" podID="f744fccd-1edc-42c6-bf53-2973623e1179" containerID="9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4" exitCode=0 Nov 01 16:42:37.729113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:37.728620 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" event={"ID":"f744fccd-1edc-42c6-bf53-2973623e1179","Type":"ContainerDied","Data":"9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4"} Nov 01 16:42:37.729113 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:37.729020 4911 scope.go:122] "RemoveContainer" containerID="9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4" Nov 01 16:42:37.730271 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:37.730241 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:42:37.730350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:37.730321 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:42:37.730398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:37.730353 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:42:38.674257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:38.674206 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:38.735951 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:38.735907 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" event={"ID":"f744fccd-1edc-42c6-bf53-2973623e1179","Type":"ContainerStarted","Data":"50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa"} Nov 01 16:42:39.729454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:39.729414 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:39.730907 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:39.730874 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:42:39.731076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:39.730931 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 16:42:40.729831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:40.729790 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:42:40.730277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:40.729874 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 16:42:40.840663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:40.840636 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:42:40.843705 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:40.843680 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:42:40.894320 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:42:40.894281 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:42:41.556090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.550489 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w"] Nov 01 16:42:41.556090 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.555023 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.571407 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.571376 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ee2b3d\"/\"kube-root-ca.crt\"" Nov 01 16:42:41.572708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.571789 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ee2b3d\"/\"openshift-service-ca.crt\"" Nov 01 16:42:41.572894 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.571894 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ee2b3d\"/\"simulator-single-main-ee2b3d-kserve-self-signed-certs\"" Nov 01 16:42:41.573007 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.572645 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ee2b3d\"/\"default-dockercfg-8q2wf\"" Nov 01 16:42:41.575255 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.575220 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w"] Nov 01 16:42:41.606650 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.606605 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-dshm\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.606982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.606956 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-home\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.607220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.607201 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-tmp-dir\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.607422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.607404 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1245723-acd9-4263-aa32-f6cddceb2039-tls-certs\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.607633 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.607610 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sb86\" (UniqueName: \"kubernetes.io/projected/e1245723-acd9-4263-aa32-f6cddceb2039-kube-api-access-5sb86\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.607846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.607827 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-model-cache\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.708807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.708766 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-dshm\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.708814 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-home\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.708834 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-tmp-dir\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.708966 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1245723-acd9-4263-aa32-f6cddceb2039-tls-certs\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.709025 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sb86\" (UniqueName: \"kubernetes.io/projected/e1245723-acd9-4263-aa32-f6cddceb2039-kube-api-access-5sb86\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.709156 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-model-cache\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.709291 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-home\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709448 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.709380 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-tmp-dir\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.709571 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.709502 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-model-cache\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.711225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.711197 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-dshm\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.711446 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.711428 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1245723-acd9-4263-aa32-f6cddceb2039-tls-certs\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.716635 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.716614 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sb86\" (UniqueName: \"kubernetes.io/projected/e1245723-acd9-4263-aa32-f6cddceb2039-kube-api-access-5sb86\") pod \"simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:41.730461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.730325 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:42:41.730461 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.730396 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 16:42:41.902640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:41.902551 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:42.207971 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:42.207943 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w"] Nov 01 16:42:42.210312 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:42:42.210272 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1245723_acd9_4263_aa32_f6cddceb2039.slice/crio-5ae40013f5bfcb45ee15d4b1efafbc8727125b28d4808c0893afce48d7f546fc WatchSource:0}: Error finding container 5ae40013f5bfcb45ee15d4b1efafbc8727125b28d4808c0893afce48d7f546fc: Status 404 returned error can't find the container with id 5ae40013f5bfcb45ee15d4b1efafbc8727125b28d4808c0893afce48d7f546fc Nov 01 16:42:42.729213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:42.729179 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:42:42.729425 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:42.729253 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 16:42:42.780433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:42.780389 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" event={"ID":"e1245723-acd9-4263-aa32-f6cddceb2039","Type":"ContainerStarted","Data":"8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b"} Nov 01 16:42:42.780850 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:42.780442 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" event={"ID":"e1245723-acd9-4263-aa32-f6cddceb2039","Type":"ContainerStarted","Data":"5ae40013f5bfcb45ee15d4b1efafbc8727125b28d4808c0893afce48d7f546fc"} Nov 01 16:42:42.799527 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:42.799477 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" podStartSLOduration=1.79946361 podStartE2EDuration="1.79946361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:42:42.798259757 +0000 UTC m=+3302.942714398" watchObservedRunningTime="2026-11-01 16:42:42.79946361 +0000 UTC m=+3302.943918250" Nov 01 16:42:42.903606 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:42.903565 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:43.729648 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:43.729608 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:42:43.729837 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:43.729692 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 01 16:42:44.733267 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:44.733234 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:44.733698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:44.733610 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:44.734353 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:44.734333 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:42:51.902839 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:51.902803 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:51.915737 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:51.915702 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:42:52.847152 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:42:52.847120 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:43:10.714729 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.714694 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk"] Nov 01 16:43:10.718815 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.718798 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:10.725083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.725054 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:43:10.725277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.725111 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:43:10.725394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.725118 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:43:10.725394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.725130 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:43:10.726671 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.726647 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk"] Nov 01 16:43:10.865120 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:43:10.865088 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:43:10.905959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.905924 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:10.905959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.905961 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcqv\" (UniqueName: \"kubernetes.io/projected/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-kube-api-access-qvcqv\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:10.906199 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:10.906052 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.007290 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.007241 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcqv\" (UniqueName: \"kubernetes.io/projected/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-kube-api-access-qvcqv\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.007483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.007306 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.007483 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.007423 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.008145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.008124 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.009734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.009708 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.016686 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.016653 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcqv\" (UniqueName: \"kubernetes.io/projected/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-kube-api-access-qvcqv\") pod \"maas-api-revoke-sub-c6d7b577-6llwk\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.040512 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.040479 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:11.331447 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.331411 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk"] Nov 01 16:43:11.332820 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:43:11.332791 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdd046a_6ea5_4ca4_87e7_7cf40a68a54e.slice/crio-0eb33d2bc85099d7ebf8fd961898156bc2c099411eb825b7b082ff3568f1fc58 WatchSource:0}: Error finding container 0eb33d2bc85099d7ebf8fd961898156bc2c099411eb825b7b082ff3568f1fc58: Status 404 returned error can't find the container with id 0eb33d2bc85099d7ebf8fd961898156bc2c099411eb825b7b082ff3568f1fc58 Nov 01 16:43:11.941393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.941295 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" event={"ID":"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e","Type":"ContainerStarted","Data":"b8a686530aa9986bcff672c1ce715b731e0c96d7824e5acca2a252dc84f1de87"} Nov 01 16:43:11.941393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.941388 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" event={"ID":"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e","Type":"ContainerStarted","Data":"0eb33d2bc85099d7ebf8fd961898156bc2c099411eb825b7b082ff3568f1fc58"} Nov 01 16:43:11.965285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:11.965220 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" podStartSLOduration=1.965202646 podStartE2EDuration="1.965202646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:43:11.960836402 +0000 UTC m=+3332.105291045" watchObservedRunningTime="2026-11-01 16:43:11.965202646 +0000 UTC m=+3332.109657288" Nov 01 16:43:12.947575 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:12.947527 4911 generic.go:356] "Generic (PLEG): container finished" podID="7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" containerID="b8a686530aa9986bcff672c1ce715b731e0c96d7824e5acca2a252dc84f1de87" exitCode=0 Nov 01 16:43:12.948057 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:12.947606 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" event={"ID":"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e","Type":"ContainerDied","Data":"b8a686530aa9986bcff672c1ce715b731e0c96d7824e5acca2a252dc84f1de87"} Nov 01 16:43:14.091870 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.091841 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:14.132665 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.132628 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-kube-api-access-qvcqv\" (UniqueName: \"kubernetes.io/projected/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-kube-api-access-qvcqv\") pod \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " Nov 01 16:43:14.132854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.132682 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-xks-ca\") pod \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " Nov 01 16:43:14.132854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.132737 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-openshift-ca\") pod \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\" (UID: \"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e\") " Nov 01 16:43:14.133164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.133131 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-openshift-ca" pod "7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" (UID: "7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:43:14.135234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.135201 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-xks-ca" pod "7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" (UID: "7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:43:14.135586 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.135555 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-kube-api-access-qvcqv" pod "7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" (UID: "7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e"). InnerVolumeSpecName "kube-api-access-qvcqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:14.233581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.233476 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcqv\" (UniqueName: \"kubernetes.io/projected/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-kube-api-access-qvcqv\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:14.233581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.233509 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:14.233581 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.233520 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:14.960030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.959995 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" Nov 01 16:43:14.960261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.959995 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk" event={"ID":"7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e","Type":"ContainerDied","Data":"0eb33d2bc85099d7ebf8fd961898156bc2c099411eb825b7b082ff3568f1fc58"} Nov 01 16:43:14.960261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:14.960138 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb33d2bc85099d7ebf8fd961898156bc2c099411eb825b7b082ff3568f1fc58" Nov 01 16:43:21.841895 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.841853 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh"] Nov 01 16:43:21.842489 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.842472 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" containerName="revoke-keys" Nov 01 16:43:21.842535 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.842491 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" containerName="revoke-keys" Nov 01 16:43:21.842600 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.842588 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" containerName="revoke-keys" Nov 01 16:43:21.846348 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.846326 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:21.851874 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.851834 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:43:21.852031 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.851908 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:43:21.853354 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.853323 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh"] Nov 01 16:43:21.907805 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.907766 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:21.908018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.907829 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:21.908018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.907986 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrt7\" (UniqueName: \"kubernetes.io/projected/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-kube-api-access-blrt7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:21.933240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.933204 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:43:21.933240 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:21.933221 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:43:22.009622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.009575 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrt7\" (UniqueName: \"kubernetes.io/projected/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-kube-api-access-blrt7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:22.009862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.009798 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:22.009862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.009857 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:22.010614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.010586 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:22.012338 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.012306 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:22.017819 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.017796 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrt7\" (UniqueName: \"kubernetes.io/projected/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-kube-api-access-blrt7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:22.163470 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.163382 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:22.439872 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:22.439848 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh"] Nov 01 16:43:22.442425 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:43:22.442394 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962c13bd_c9ee_40a0_a8cf_8fd8829b0cb7.slice/crio-e1880a2b528e3ccf8014cd7892c534069e1c84be597cad4a09068adc52b97c92 WatchSource:0}: Error finding container e1880a2b528e3ccf8014cd7892c534069e1c84be597cad4a09068adc52b97c92: Status 404 returned error can't find the container with id e1880a2b528e3ccf8014cd7892c534069e1c84be597cad4a09068adc52b97c92 Nov 01 16:43:23.010248 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:23.010200 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" event={"ID":"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7","Type":"ContainerStarted","Data":"70fcb3ffdf9f96e3ddf5ade0bb204c7bff96b9c0d6ba6dab406247db1e61e58d"} Nov 01 16:43:23.010777 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:23.010272 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" event={"ID":"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7","Type":"ContainerStarted","Data":"e1880a2b528e3ccf8014cd7892c534069e1c84be597cad4a09068adc52b97c92"} Nov 01 16:43:23.029399 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:23.029299 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" podStartSLOduration=2.029272709 podStartE2EDuration="2.029272709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:43:23.026618123 +0000 UTC m=+3343.171072761" watchObservedRunningTime="2026-11-01 16:43:23.029272709 +0000 UTC m=+3343.173727349" Nov 01 16:43:24.016793 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:24.016759 4911 generic.go:356] "Generic (PLEG): container finished" podID="962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" containerID="70fcb3ffdf9f96e3ddf5ade0bb204c7bff96b9c0d6ba6dab406247db1e61e58d" exitCode=0 Nov 01 16:43:24.017247 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:24.016832 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" event={"ID":"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7","Type":"ContainerDied","Data":"70fcb3ffdf9f96e3ddf5ade0bb204c7bff96b9c0d6ba6dab406247db1e61e58d"} Nov 01 16:43:25.165185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.165155 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:25.238311 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.238267 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-kube-api-access-blrt7\" (UniqueName: \"kubernetes.io/projected/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-kube-api-access-blrt7\") pod \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " Nov 01 16:43:25.238524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.238381 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-openshift-ca\") pod \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " Nov 01 16:43:25.238524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.238432 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-xks-ca\") pod \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\" (UID: \"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7\") " Nov 01 16:43:25.238812 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.238786 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-openshift-ca" pod "962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" (UID: "962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:43:25.240473 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.240449 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-xks-ca" pod "962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" (UID: "962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:43:25.240542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.240516 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-kube-api-access-blrt7" pod "962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" (UID: "962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7"). InnerVolumeSpecName "kube-api-access-blrt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:25.340225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.340138 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:25.340225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.340170 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:25.340225 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:25.340182 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blrt7\" (UniqueName: \"kubernetes.io/projected/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7-kube-api-access-blrt7\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:26.027772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:26.027741 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" Nov 01 16:43:26.027772 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:26.027759 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh" event={"ID":"962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7","Type":"ContainerDied","Data":"e1880a2b528e3ccf8014cd7892c534069e1c84be597cad4a09068adc52b97c92"} Nov 01 16:43:26.027983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:26.027801 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1880a2b528e3ccf8014cd7892c534069e1c84be597cad4a09068adc52b97c92" Nov 01 16:43:26.621458 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:26.621409 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w"] Nov 01 16:43:32.810018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:32.809980 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh"] Nov 01 16:43:32.813523 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:32.813496 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee2b3d-q9rkh"] Nov 01 16:43:33.965001 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:33.964963 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" path="/var/lib/kubelet/pods/962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7/volumes" Nov 01 16:43:39.437593 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:39.437551 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5"] Nov 01 16:43:39.438246 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:39.437794 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" containerID="cri-o://50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa" gracePeriod=30 Nov 01 16:43:40.864564 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:43:40.864528 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:43:41.641734 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.641690 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" podUID="e1245723-acd9-4263-aa32-f6cddceb2039" containerName="main" containerID="cri-o://8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b" gracePeriod=45 Nov 01 16:43:41.885335 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.885308 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:43:41.991628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.991590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-model-cache\") pod \"e1245723-acd9-4263-aa32-f6cddceb2039\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " Nov 01 16:43:41.991814 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.991798 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-dshm\") pod \"e1245723-acd9-4263-aa32-f6cddceb2039\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " Nov 01 16:43:41.991890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.991853 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-model-cache" pod "e1245723-acd9-4263-aa32-f6cddceb2039" (UID: "e1245723-acd9-4263-aa32-f6cddceb2039"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:41.991959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.991891 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1245723-acd9-4263-aa32-f6cddceb2039-kube-api-access-5sb86\" (UniqueName: \"kubernetes.io/projected/e1245723-acd9-4263-aa32-f6cddceb2039-kube-api-access-5sb86\") pod \"e1245723-acd9-4263-aa32-f6cddceb2039\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " Nov 01 16:43:41.991959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.991941 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-home\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-home\") pod \"e1245723-acd9-4263-aa32-f6cddceb2039\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " Nov 01 16:43:41.992086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.992011 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-tmp-dir\") pod \"e1245723-acd9-4263-aa32-f6cddceb2039\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " Nov 01 16:43:41.992086 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.992060 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1245723-acd9-4263-aa32-f6cddceb2039-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1245723-acd9-4263-aa32-f6cddceb2039-tls-certs\") pod \"e1245723-acd9-4263-aa32-f6cddceb2039\" (UID: \"e1245723-acd9-4263-aa32-f6cddceb2039\") " Nov 01 16:43:41.992200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.992176 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-home" pod "e1245723-acd9-4263-aa32-f6cddceb2039" (UID: "e1245723-acd9-4263-aa32-f6cddceb2039"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:41.992311 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.992280 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-tmp-dir" pod "e1245723-acd9-4263-aa32-f6cddceb2039" (UID: "e1245723-acd9-4263-aa32-f6cddceb2039"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:41.992446 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.992434 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:41.992518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.992449 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:41.992518 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.992459 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:41.994219 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.994194 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-dshm" pod "e1245723-acd9-4263-aa32-f6cddceb2039" (UID: "e1245723-acd9-4263-aa32-f6cddceb2039"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:41.994459 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.994436 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1245723-acd9-4263-aa32-f6cddceb2039-tls-certs" pod "e1245723-acd9-4263-aa32-f6cddceb2039" (UID: "e1245723-acd9-4263-aa32-f6cddceb2039"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:43:41.994536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:41.994472 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1245723-acd9-4263-aa32-f6cddceb2039-kube-api-access-5sb86" pod "e1245723-acd9-4263-aa32-f6cddceb2039" (UID: "e1245723-acd9-4263-aa32-f6cddceb2039"). InnerVolumeSpecName "kube-api-access-5sb86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:42.093401 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.093356 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1245723-acd9-4263-aa32-f6cddceb2039-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:42.093401 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.093395 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sb86\" (UniqueName: \"kubernetes.io/projected/e1245723-acd9-4263-aa32-f6cddceb2039-kube-api-access-5sb86\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:42.093401 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.093406 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1245723-acd9-4263-aa32-f6cddceb2039-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:42.113939 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.113907 4911 generic.go:356] "Generic (PLEG): container finished" podID="e1245723-acd9-4263-aa32-f6cddceb2039" containerID="8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b" exitCode=0 Nov 01 16:43:42.114100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.113954 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" Nov 01 16:43:42.114100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.113995 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" event={"ID":"e1245723-acd9-4263-aa32-f6cddceb2039","Type":"ContainerDied","Data":"8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b"} Nov 01 16:43:42.114100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.114053 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" event={"ID":"e1245723-acd9-4263-aa32-f6cddceb2039","Type":"ContainerDied","Data":"5ae40013f5bfcb45ee15d4b1efafbc8727125b28d4808c0893afce48d7f546fc"} Nov 01 16:43:42.114100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.114077 4911 scope.go:122] "RemoveContainer" containerID="8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b" Nov 01 16:43:42.124122 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.124100 4911 scope.go:122] "RemoveContainer" containerID="8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b" Nov 01 16:43:42.124388 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:43:42.124367 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b\": container with ID starting with 8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b not found: ID does not exist" containerID="8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b" Nov 01 16:43:42.124439 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.124396 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b"} err="failed to get container status \"8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b\": rpc error: code = NotFound desc = could not find container \"8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b\": container with ID starting with 8de42ab86439bc1ca2448eb3f9f3c233299416c19dbfe07a9ef316f1eb21c75b not found: ID does not exist" Nov 01 16:43:42.137054 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.137013 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w"] Nov 01 16:43:42.141241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.141218 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w"] Nov 01 16:43:42.836652 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:42.836611 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-ee2b3d/simulator-single-main-ee2b3d-kserve-74cdff9f55-mzm8w" podUID="e1245723-acd9-4263-aa32-f6cddceb2039" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded" Nov 01 16:43:43.964269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:43.964231 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1245723-acd9-4263-aa32-f6cddceb2039" path="/var/lib/kubelet/pods/e1245723-acd9-4263-aa32-f6cddceb2039/volumes" Nov 01 16:43:44.699252 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.699225 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:43:44.817689 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817645 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-socket\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.817902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817734 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f744fccd-1edc-42c6-bf53-2973623e1179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f744fccd-1edc-42c6-bf53-2973623e1179-istiod-ca-cert\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.817902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817776 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-istio-token\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-istio-token\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.817902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817819 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-kube-api-access-gxsch\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-kube-api-access-gxsch\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.817902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817853 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-credential-socket\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.817902 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817879 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f744fccd-1edc-42c6-bf53-2973623e1179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f744fccd-1edc-42c6-bf53-2973623e1179-istio-podinfo\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.818232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817956 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-socket" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:44.818232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.817963 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-certs\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.818232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818082 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-data\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.818232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818114 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-envoy\") pod \"f744fccd-1edc-42c6-bf53-2973623e1179\" (UID: \"f744fccd-1edc-42c6-bf53-2973623e1179\") " Nov 01 16:43:44.818436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818244 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-credential-socket" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:44.818436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818302 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f744fccd-1edc-42c6-bf53-2973623e1179-istiod-ca-cert" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:43:44.818436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818418 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-data" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:44.818576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818444 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f744fccd-1edc-42c6-bf53-2973623e1179-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.818576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818467 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.818576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818482 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.818576 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.818485 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-certs" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:44.820145 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.820115 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f744fccd-1edc-42c6-bf53-2973623e1179-istio-podinfo" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:43:44.820285 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.820237 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-envoy" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:43:44.820539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.820524 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-istio-token" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:44.820588 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.820571 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-kube-api-access-gxsch" pod "f744fccd-1edc-42c6-bf53-2973623e1179" (UID: "f744fccd-1edc-42c6-bf53-2973623e1179"). InnerVolumeSpecName "kube-api-access-gxsch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:44.919098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.919032 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.919098 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.919093 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.919323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.919108 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.919323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.919123 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxsch\" (UniqueName: \"kubernetes.io/projected/f744fccd-1edc-42c6-bf53-2973623e1179-kube-api-access-gxsch\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.919323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.919138 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f744fccd-1edc-42c6-bf53-2973623e1179-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.919323 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:44.919150 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f744fccd-1edc-42c6-bf53-2973623e1179-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:45.131340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.131247 4911 generic.go:356] "Generic (PLEG): container finished" podID="f744fccd-1edc-42c6-bf53-2973623e1179" containerID="50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa" exitCode=0 Nov 01 16:43:45.131340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.131297 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" Nov 01 16:43:45.131340 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.131319 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" event={"ID":"f744fccd-1edc-42c6-bf53-2973623e1179","Type":"ContainerDied","Data":"50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa"} Nov 01 16:43:45.131950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.131389 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5" event={"ID":"f744fccd-1edc-42c6-bf53-2973623e1179","Type":"ContainerDied","Data":"e523f840a11f15eda49ac802c32d43deaccff9c684f23bfc8b005ba789714aab"} Nov 01 16:43:45.131950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.131420 4911 scope.go:122] "RemoveContainer" containerID="50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa" Nov 01 16:43:45.145858 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.145836 4911 scope.go:122] "RemoveContainer" containerID="9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4" Nov 01 16:43:45.155432 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.155406 4911 scope.go:122] "RemoveContainer" containerID="50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa" Nov 01 16:43:45.155817 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:43:45.155792 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa\": container with ID starting with 50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa not found: ID does not exist" containerID="50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa" Nov 01 16:43:45.156034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.156002 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa"} err="failed to get container status \"50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa\": rpc error: code = NotFound desc = could not find container \"50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa\": container with ID starting with 50c37daf890b527dfcdeb9db766096315669fbdbce6cfc4b37839a7fe4b2fcfa not found: ID does not exist" Nov 01 16:43:45.156034 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.156034 4911 scope.go:122] "RemoveContainer" containerID="9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4" Nov 01 16:43:45.156364 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:43:45.156343 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4\": container with ID starting with 9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4 not found: ID does not exist" containerID="9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4" Nov 01 16:43:45.156433 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.156376 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4"} err="failed to get container status \"9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4\": rpc error: code = NotFound desc = could not find container \"9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4\": container with ID starting with 9c5a076b3892cd6007c8d897a898accfabe446c71f7c76b1c5d1799c77fe6ef4 not found: ID does not exist" Nov 01 16:43:45.158436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.158411 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5"] Nov 01 16:43:45.161942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.161919 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee2b3d-openshift-default-75656657ccjfrc5"] Nov 01 16:43:45.965516 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:43:45.965483 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" path="/var/lib/kubelet/pods/f744fccd-1edc-42c6-bf53-2973623e1179/volumes" Nov 01 16:44:10.865695 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:44:10.865647 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:44:40.865583 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:44:40.865543 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:45:10.866648 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:45:10.866610 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:45:40.934090 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:45:40.934029 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:45:56.731500 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.731443 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8"] Nov 01 16:45:56.732087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732028 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e1245723-acd9-4263-aa32-f6cddceb2039" containerName="main" Nov 01 16:45:56.732087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732066 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1245723-acd9-4263-aa32-f6cddceb2039" containerName="main" Nov 01 16:45:56.732249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732101 4911 cpu_manager.go:414] "containerMap: removing container" podUID="962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" containerName="revoke-keys" Nov 01 16:45:56.732249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732108 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" containerName="revoke-keys" Nov 01 16:45:56.732249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732118 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" Nov 01 16:45:56.732249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732123 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" Nov 01 16:45:56.732249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732147 4911 cpu_manager.go:414] "containerMap: removing container" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" Nov 01 16:45:56.732249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732155 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" Nov 01 16:45:56.732249 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732247 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" Nov 01 16:45:56.732634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732268 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1245723-acd9-4263-aa32-f6cddceb2039" containerName="main" Nov 01 16:45:56.732634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.732282 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="962c13bd-c9ee-40a0-a8cf-8fd8829b0cb7" containerName="revoke-keys" Nov 01 16:45:56.735835 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.735807 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.754702 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.754582 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-230d0a-openshift-default-dockercfg-xchh9\"" Nov 01 16:45:56.757834 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.757806 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8"] Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777344 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e18849-e895-417d-8978-f9646f997c3c-istiod-ca-cert\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777411 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-credential-socket\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777462 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-certs\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777483 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e18849-e895-417d-8978-f9646f997c3c-istio-podinfo\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777509 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-istio-token\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777541 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-data\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777560 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-envoy\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777586 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-socket\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.778108 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.777615 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xks\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-kube-api-access-m4xks\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878393 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878352 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e18849-e895-417d-8978-f9646f997c3c-istiod-ca-cert\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878407 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-credential-socket\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878444 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-certs\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878464 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e18849-e895-417d-8978-f9646f997c3c-istio-podinfo\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878487 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-istio-token\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878522 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-data\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878549 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-envoy\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.878589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878587 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-socket\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.879056 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.878617 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xks\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-kube-api-access-m4xks\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.879305 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.879273 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-credential-socket\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.879403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.879305 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e18849-e895-417d-8978-f9646f997c3c-istiod-ca-cert\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.879403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.879309 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-certs\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.879528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.879478 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-data\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.879528 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.879497 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-socket\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.880979 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.880957 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-envoy\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.881454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.881435 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e18849-e895-417d-8978-f9646f997c3c-istio-podinfo\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.886818 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.886795 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-istio-token\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:56.886975 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:56.886954 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xks\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-kube-api-access-m4xks\") pod \"e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:57.073159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.073122 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:57.353121 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.353084 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8"] Nov 01 16:45:57.355285 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:45:57.355247 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e18849_e895_417d_8978_f9646f997c3c.slice/crio-0ad59862aaca04fbddc2417385916b5e0de9ba58e12c4985d56e7b5f0420247b WatchSource:0}: Error finding container 0ad59862aaca04fbddc2417385916b5e0de9ba58e12c4985d56e7b5f0420247b: Status 404 returned error can't find the container with id 0ad59862aaca04fbddc2417385916b5e0de9ba58e12c4985d56e7b5f0420247b Nov 01 16:45:57.357906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.357871 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:45:57.357994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.357940 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:45:57.357994 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.357969 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:45:57.916088 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.916013 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" event={"ID":"d5e18849-e895-417d-8978-f9646f997c3c","Type":"ContainerStarted","Data":"9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9"} Nov 01 16:45:57.916477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.916126 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" event={"ID":"d5e18849-e895-417d-8978-f9646f997c3c","Type":"ContainerStarted","Data":"0ad59862aaca04fbddc2417385916b5e0de9ba58e12c4985d56e7b5f0420247b"} Nov 01 16:45:57.937993 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:57.937935 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podStartSLOduration=1.937918209 podStartE2EDuration="1.937918209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:45:57.937731803 +0000 UTC m=+3498.082186445" watchObservedRunningTime="2026-11-01 16:45:57.937918209 +0000 UTC m=+3498.082372851" Nov 01 16:45:58.073364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:58.073329 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:58.078378 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:58.078349 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:58.921967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:58.921931 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:45:58.922930 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:45:58.922907 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:46:10.865980 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:46:10.865947 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:46:13.923144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:13.923103 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:46:13.923554 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:13.923175 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 16:46:14.023913 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:14.023876 4911 generic.go:356] "Generic (PLEG): container finished" podID="d5e18849-e895-417d-8978-f9646f997c3c" containerID="9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9" exitCode=0 Nov 01 16:46:14.024102 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:14.023924 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" event={"ID":"d5e18849-e895-417d-8978-f9646f997c3c","Type":"ContainerDied","Data":"9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9"} Nov 01 16:46:14.024315 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:14.024300 4911 scope.go:122] "RemoveContainer" containerID="9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9" Nov 01 16:46:14.025539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:14.025507 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:46:14.025626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:14.025581 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:46:14.025626 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:14.025616 4911 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:46:15.030810 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:15.030772 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" event={"ID":"d5e18849-e895-417d-8978-f9646f997c3c","Type":"ContainerStarted","Data":"48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e"} Nov 01 16:46:15.074241 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:15.074200 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:46:15.074521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:15.074490 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:46:15.074622 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:15.074561 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 16:46:16.074640 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:16.074599 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:46:16.075831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:16.075300 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 16:46:16.988295 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:16.988259 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc"] Nov 01 16:46:16.988757 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:16.988743 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="f744fccd-1edc-42c6-bf53-2973623e1179" containerName="istio-proxy" Nov 01 16:46:16.992009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:16.991990 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:16.998364 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:16.998336 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-230d0a\"/\"kube-root-ca.crt\"" Nov 01 16:46:17.001076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.001018 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-230d0a\"/\"openshift-service-ca.crt\"" Nov 01 16:46:17.001680 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.001663 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-230d0a\"/\"default-dockercfg-jg6j9\"" Nov 01 16:46:17.001795 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.001775 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-230d0a\"/\"simulator-single-main-230d0a-kserve-self-signed-certs\"" Nov 01 16:46:17.003266 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.003241 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc"] Nov 01 16:46:17.082118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.082072 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:46:17.082663 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.082167 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 16:46:17.097607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.097575 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/561bbf92-1f62-4537-99a1-f7154dc86a22-tls-certs\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.097754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.097622 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-dshm\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.097754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.097692 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-home\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.097754 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.097716 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-model-cache\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.097867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.097802 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-tmp-dir\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.097867 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.097853 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9lmd\" (UniqueName: \"kubernetes.io/projected/561bbf92-1f62-4537-99a1-f7154dc86a22-kube-api-access-g9lmd\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.198590 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.198541 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/561bbf92-1f62-4537-99a1-f7154dc86a22-tls-certs\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.198811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.198609 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-dshm\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.198811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.198640 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-home\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.198811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.198662 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-model-cache\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.198811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.198707 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-tmp-dir\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.198811 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.198724 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9lmd\" (UniqueName: \"kubernetes.io/projected/561bbf92-1f62-4537-99a1-f7154dc86a22-kube-api-access-g9lmd\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.199890 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.199859 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-home\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.200201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.200170 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-model-cache\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.200488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.200468 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-tmp-dir\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.202381 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.202353 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/561bbf92-1f62-4537-99a1-f7154dc86a22-tls-certs\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.204291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.204265 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-dshm\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.213441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.213415 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9lmd\" (UniqueName: \"kubernetes.io/projected/561bbf92-1f62-4537-99a1-f7154dc86a22-kube-api-access-g9lmd\") pod \"simulator-single-main-230d0a-kserve-9ff784f78-mgsdc\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.308651 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.308567 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:17.615494 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:17.615461 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc"] Nov 01 16:46:17.616796 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:46:17.616765 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod561bbf92_1f62_4537_99a1_f7154dc86a22.slice/crio-81d19f85a28add0c987eab9e78ddd6252827790665da3ae500371a93ee4a7787 WatchSource:0}: Error finding container 81d19f85a28add0c987eab9e78ddd6252827790665da3ae500371a93ee4a7787: Status 404 returned error can't find the container with id 81d19f85a28add0c987eab9e78ddd6252827790665da3ae500371a93ee4a7787 Nov 01 16:46:18.069367 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:18.069328 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" event={"ID":"561bbf92-1f62-4537-99a1-f7154dc86a22","Type":"ContainerStarted","Data":"8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c"} Nov 01 16:46:18.069687 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:18.069651 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" event={"ID":"561bbf92-1f62-4537-99a1-f7154dc86a22","Type":"ContainerStarted","Data":"81d19f85a28add0c987eab9e78ddd6252827790665da3ae500371a93ee4a7787"} Nov 01 16:46:18.075466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:18.075441 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:46:18.076083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:18.076029 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 16:46:18.087677 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:18.087630 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" podStartSLOduration=2.087617633 podStartE2EDuration="2.087617633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:46:18.08688654 +0000 UTC m=+3518.231341173" watchObservedRunningTime="2026-11-01 16:46:18.087617633 +0000 UTC m=+3518.232072275" Nov 01 16:46:18.308842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:18.308810 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:19.077064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:19.076885 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 01 16:46:19.077064 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:19.076945 4911 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 01 16:46:20.508015 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:20.507971 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:46:20.508482 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:20.508417 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:46:20.509465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:20.509442 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:46:27.308764 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:27.308715 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:27.321863 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:27.321838 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:28.142799 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:28.142767 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:46:40.873643 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:46:40.873598 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:46:47.207394 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.207360 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj"] Nov 01 16:46:47.211002 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.210984 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.217862 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.217836 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:46:47.218025 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.217875 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:46:47.218139 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.217985 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:46:47.219506 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.219485 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:46:47.222492 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.222465 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj"] Nov 01 16:46:47.306644 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.306606 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tgt\" (UniqueName: \"kubernetes.io/projected/7da37491-0107-4491-a065-77c2c865eebf-kube-api-access-76tgt\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.306824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.306667 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da37491-0107-4491-a065-77c2c865eebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.306824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.306763 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da37491-0107-4491-a065-77c2c865eebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.408149 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.408112 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da37491-0107-4491-a065-77c2c865eebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.408361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.408163 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da37491-0107-4491-a065-77c2c865eebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.408361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.408293 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tgt\" (UniqueName: \"kubernetes.io/projected/7da37491-0107-4491-a065-77c2c865eebf-kube-api-access-76tgt\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.408861 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.408835 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da37491-0107-4491-a065-77c2c865eebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.410573 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.410553 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da37491-0107-4491-a065-77c2c865eebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.418557 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.418531 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tgt\" (UniqueName: \"kubernetes.io/projected/7da37491-0107-4491-a065-77c2c865eebf-kube-api-access-76tgt\") pod \"maas-api-revoke-sub-33384c4a-6ksgj\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.526691 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.526660 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:47.804234 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:47.804205 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj"] Nov 01 16:46:47.806321 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:46:47.806294 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da37491_0107_4491_a065_77c2c865eebf.slice/crio-121b923d6532b0dba4fea5c8c84094d32528eb9eced5a6546d75112ba6dc16d0 WatchSource:0}: Error finding container 121b923d6532b0dba4fea5c8c84094d32528eb9eced5a6546d75112ba6dc16d0: Status 404 returned error can't find the container with id 121b923d6532b0dba4fea5c8c84094d32528eb9eced5a6546d75112ba6dc16d0 Nov 01 16:46:48.241885 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:48.241837 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" event={"ID":"7da37491-0107-4491-a065-77c2c865eebf","Type":"ContainerStarted","Data":"93e48286d0c08529d0cda7cce973642d34e895f38de6a06c9bbf04ec9ee09d62"} Nov 01 16:46:48.242417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:48.241904 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" event={"ID":"7da37491-0107-4491-a065-77c2c865eebf","Type":"ContainerStarted","Data":"121b923d6532b0dba4fea5c8c84094d32528eb9eced5a6546d75112ba6dc16d0"} Nov 01 16:46:48.258860 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:48.258808 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" podStartSLOduration=1.258794408 podStartE2EDuration="1.258794408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:46:48.257899613 +0000 UTC m=+3548.402354255" watchObservedRunningTime="2026-11-01 16:46:48.258794408 +0000 UTC m=+3548.403249049" Nov 01 16:46:49.249150 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:49.249106 4911 generic.go:356] "Generic (PLEG): container finished" podID="7da37491-0107-4491-a065-77c2c865eebf" containerID="93e48286d0c08529d0cda7cce973642d34e895f38de6a06c9bbf04ec9ee09d62" exitCode=0 Nov 01 16:46:49.249634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:49.249155 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" event={"ID":"7da37491-0107-4491-a065-77c2c865eebf","Type":"ContainerDied","Data":"93e48286d0c08529d0cda7cce973642d34e895f38de6a06c9bbf04ec9ee09d62"} Nov 01 16:46:50.396724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.396694 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:50.436827 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.436795 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da37491-0107-4491-a065-77c2c865eebf-kube-api-access-76tgt\" (UniqueName: \"kubernetes.io/projected/7da37491-0107-4491-a065-77c2c865eebf-kube-api-access-76tgt\") pod \"7da37491-0107-4491-a065-77c2c865eebf\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " Nov 01 16:46:50.437020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.436858 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7da37491-0107-4491-a065-77c2c865eebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da37491-0107-4491-a065-77c2c865eebf-maas-api-xks-ca\") pod \"7da37491-0107-4491-a065-77c2c865eebf\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " Nov 01 16:46:50.437020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.436974 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da37491-0107-4491-a065-77c2c865eebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da37491-0107-4491-a065-77c2c865eebf-maas-api-openshift-ca\") pod \"7da37491-0107-4491-a065-77c2c865eebf\" (UID: \"7da37491-0107-4491-a065-77c2c865eebf\") " Nov 01 16:46:50.437397 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.437365 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da37491-0107-4491-a065-77c2c865eebf-maas-api-openshift-ca" pod "7da37491-0107-4491-a065-77c2c865eebf" (UID: "7da37491-0107-4491-a065-77c2c865eebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:46:50.441358 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.441334 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da37491-0107-4491-a065-77c2c865eebf-maas-api-xks-ca" pod "7da37491-0107-4491-a065-77c2c865eebf" (UID: "7da37491-0107-4491-a065-77c2c865eebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:46:50.441544 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.441516 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da37491-0107-4491-a065-77c2c865eebf-kube-api-access-76tgt" pod "7da37491-0107-4491-a065-77c2c865eebf" (UID: "7da37491-0107-4491-a065-77c2c865eebf"). InnerVolumeSpecName "kube-api-access-76tgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:46:50.538846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.538764 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da37491-0107-4491-a065-77c2c865eebf-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:46:50.538846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.538799 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76tgt\" (UniqueName: \"kubernetes.io/projected/7da37491-0107-4491-a065-77c2c865eebf-kube-api-access-76tgt\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:46:50.538846 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:50.538815 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da37491-0107-4491-a065-77c2c865eebf-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:46:51.261901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:51.261866 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" Nov 01 16:46:51.261901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:51.261887 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj" event={"ID":"7da37491-0107-4491-a065-77c2c865eebf","Type":"ContainerDied","Data":"121b923d6532b0dba4fea5c8c84094d32528eb9eced5a6546d75112ba6dc16d0"} Nov 01 16:46:51.262156 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:51.261964 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="121b923d6532b0dba4fea5c8c84094d32528eb9eced5a6546d75112ba6dc16d0" Nov 01 16:46:58.072967 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.072924 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d"] Nov 01 16:46:58.073685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.073656 4911 cpu_manager.go:414] "containerMap: removing container" podUID="7da37491-0107-4491-a065-77c2c865eebf" containerName="revoke-keys" Nov 01 16:46:58.073685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.073685 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da37491-0107-4491-a065-77c2c865eebf" containerName="revoke-keys" Nov 01 16:46:58.073918 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.073807 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da37491-0107-4491-a065-77c2c865eebf" containerName="revoke-keys" Nov 01 16:46:58.077369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.077347 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.082543 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.082514 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:46:58.082684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.082542 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:46:58.082684 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.082630 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:46:58.082903 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.082868 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d"] Nov 01 16:46:58.083339 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.083322 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:46:58.219845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.219809 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.220022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.219864 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.220022 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.219931 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz2fd\" (UniqueName: \"kubernetes.io/projected/1541d13a-91ee-4f67-8e36-4f520e011b15-kube-api-access-vz2fd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.321291 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.321254 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.321508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.321307 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.321508 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.321327 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz2fd\" (UniqueName: \"kubernetes.io/projected/1541d13a-91ee-4f67-8e36-4f520e011b15-kube-api-access-vz2fd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.322005 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.321978 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.323828 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.323777 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.330597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.330573 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz2fd\" (UniqueName: \"kubernetes.io/projected/1541d13a-91ee-4f67-8e36-4f520e011b15-kube-api-access-vz2fd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.393698 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.393665 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:46:58.668265 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:58.668237 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d"] Nov 01 16:46:58.670008 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:46:58.669978 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1541d13a_91ee_4f67_8e36_4f520e011b15.slice/crio-4ec5b8545af6a74be0f4ee5befe1058131046d1f55c556373adf8ab44707b070 WatchSource:0}: Error finding container 4ec5b8545af6a74be0f4ee5befe1058131046d1f55c556373adf8ab44707b070: Status 404 returned error can't find the container with id 4ec5b8545af6a74be0f4ee5befe1058131046d1f55c556373adf8ab44707b070 Nov 01 16:46:59.307298 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:59.307258 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" event={"ID":"1541d13a-91ee-4f67-8e36-4f520e011b15","Type":"ContainerStarted","Data":"94dc98ac976f1e75ec9e00ab7af20e5c6ab2e4826a21e8815d34f7edb7315ad2"} Nov 01 16:46:59.307735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:59.307312 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" event={"ID":"1541d13a-91ee-4f67-8e36-4f520e011b15","Type":"ContainerStarted","Data":"4ec5b8545af6a74be0f4ee5befe1058131046d1f55c556373adf8ab44707b070"} Nov 01 16:46:59.325610 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:46:59.325556 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" podStartSLOduration=1.325542854 podStartE2EDuration="1.325542854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:46:59.322509513 +0000 UTC m=+3559.466964155" watchObservedRunningTime="2026-11-01 16:46:59.325542854 +0000 UTC m=+3559.469997494" Nov 01 16:47:00.313789 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:00.313752 4911 generic.go:356] "Generic (PLEG): container finished" podID="1541d13a-91ee-4f67-8e36-4f520e011b15" containerID="94dc98ac976f1e75ec9e00ab7af20e5c6ab2e4826a21e8815d34f7edb7315ad2" exitCode=0 Nov 01 16:47:00.314220 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:00.313806 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" event={"ID":"1541d13a-91ee-4f67-8e36-4f520e011b15","Type":"ContainerDied","Data":"94dc98ac976f1e75ec9e00ab7af20e5c6ab2e4826a21e8815d34f7edb7315ad2"} Nov 01 16:47:01.470201 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.470169 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:47:01.553155 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.553114 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-xks-ca\") pod \"1541d13a-91ee-4f67-8e36-4f520e011b15\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " Nov 01 16:47:01.553384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.553227 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1541d13a-91ee-4f67-8e36-4f520e011b15-kube-api-access-vz2fd\" (UniqueName: \"kubernetes.io/projected/1541d13a-91ee-4f67-8e36-4f520e011b15-kube-api-access-vz2fd\") pod \"1541d13a-91ee-4f67-8e36-4f520e011b15\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " Nov 01 16:47:01.553384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.553288 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-openshift-ca\") pod \"1541d13a-91ee-4f67-8e36-4f520e011b15\" (UID: \"1541d13a-91ee-4f67-8e36-4f520e011b15\") " Nov 01 16:47:01.553849 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.553821 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-openshift-ca" pod "1541d13a-91ee-4f67-8e36-4f520e011b15" (UID: "1541d13a-91ee-4f67-8e36-4f520e011b15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:47:01.555519 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.555491 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-xks-ca" pod "1541d13a-91ee-4f67-8e36-4f520e011b15" (UID: "1541d13a-91ee-4f67-8e36-4f520e011b15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:47:01.555770 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.555744 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1541d13a-91ee-4f67-8e36-4f520e011b15-kube-api-access-vz2fd" pod "1541d13a-91ee-4f67-8e36-4f520e011b15" (UID: "1541d13a-91ee-4f67-8e36-4f520e011b15"). InnerVolumeSpecName "kube-api-access-vz2fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:01.654631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.654538 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-openshift-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:01.654631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.654574 4911 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1541d13a-91ee-4f67-8e36-4f520e011b15-maas-api-xks-ca\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:01.654631 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:01.654584 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz2fd\" (UniqueName: \"kubernetes.io/projected/1541d13a-91ee-4f67-8e36-4f520e011b15-kube-api-access-vz2fd\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:02.326232 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:02.326194 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" Nov 01 16:47:02.326418 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:02.326232 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d" event={"ID":"1541d13a-91ee-4f67-8e36-4f520e011b15","Type":"ContainerDied","Data":"4ec5b8545af6a74be0f4ee5befe1058131046d1f55c556373adf8ab44707b070"} Nov 01 16:47:02.326418 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:02.326281 4911 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ec5b8545af6a74be0f4ee5befe1058131046d1f55c556373adf8ab44707b070" Nov 01 16:47:02.889100 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:02.889057 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc"] Nov 01 16:47:09.196766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:09.196722 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d"] Nov 01 16:47:09.201369 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:09.201340 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-230d0a-k245d"] Nov 01 16:47:09.965144 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:09.965103 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1541d13a-91ee-4f67-8e36-4f520e011b15" path="/var/lib/kubelet/pods/1541d13a-91ee-4f67-8e36-4f520e011b15/volumes" Nov 01 16:47:10.867117 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:47:10.867080 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:47:15.824491 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:15.824439 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8"] Nov 01 16:47:15.824959 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:15.824744 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" containerID="cri-o://48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e" gracePeriod=30 Nov 01 16:47:17.909485 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:17.909437 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" podUID="561bbf92-1f62-4537-99a1-f7154dc86a22" containerName="main" containerID="cri-o://8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c" gracePeriod=45 Nov 01 16:47:18.174901 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.174838 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216138 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561bbf92-1f62-4537-99a1-f7154dc86a22-tls-certs\" (UniqueName: \"kubernetes.io/secret/561bbf92-1f62-4537-99a1-f7154dc86a22-tls-certs\") pod \"561bbf92-1f62-4537-99a1-f7154dc86a22\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216228 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-model-cache\") pod \"561bbf92-1f62-4537-99a1-f7154dc86a22\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216274 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-dshm\") pod \"561bbf92-1f62-4537-99a1-f7154dc86a22\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216410 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-home\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-home\") pod \"561bbf92-1f62-4537-99a1-f7154dc86a22\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216513 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561bbf92-1f62-4537-99a1-f7154dc86a22-kube-api-access-g9lmd\" (UniqueName: \"kubernetes.io/projected/561bbf92-1f62-4537-99a1-f7154dc86a22-kube-api-access-g9lmd\") pod \"561bbf92-1f62-4537-99a1-f7154dc86a22\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216553 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-tmp-dir\") pod \"561bbf92-1f62-4537-99a1-f7154dc86a22\" (UID: \"561bbf92-1f62-4537-99a1-f7154dc86a22\") " Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216856 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-model-cache" pod "561bbf92-1f62-4537-99a1-f7154dc86a22" (UID: "561bbf92-1f62-4537-99a1-f7154dc86a22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:18.217116 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.216968 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:18.217808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.217219 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-home" pod "561bbf92-1f62-4537-99a1-f7154dc86a22" (UID: "561bbf92-1f62-4537-99a1-f7154dc86a22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:18.218012 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.217946 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-tmp-dir" pod "561bbf92-1f62-4537-99a1-f7154dc86a22" (UID: "561bbf92-1f62-4537-99a1-f7154dc86a22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:18.219233 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.219204 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561bbf92-1f62-4537-99a1-f7154dc86a22-tls-certs" pod "561bbf92-1f62-4537-99a1-f7154dc86a22" (UID: "561bbf92-1f62-4537-99a1-f7154dc86a22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:47:18.219350 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.219269 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-dshm" pod "561bbf92-1f62-4537-99a1-f7154dc86a22" (UID: "561bbf92-1f62-4537-99a1-f7154dc86a22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:18.219679 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.219659 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561bbf92-1f62-4537-99a1-f7154dc86a22-kube-api-access-g9lmd" pod "561bbf92-1f62-4537-99a1-f7154dc86a22" (UID: "561bbf92-1f62-4537-99a1-f7154dc86a22"). InnerVolumeSpecName "kube-api-access-g9lmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:18.317809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.317766 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:18.317809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.317808 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/561bbf92-1f62-4537-99a1-f7154dc86a22-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:18.318233 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.317822 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:18.318233 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.317838 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/561bbf92-1f62-4537-99a1-f7154dc86a22-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:18.318233 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.317852 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9lmd\" (UniqueName: \"kubernetes.io/projected/561bbf92-1f62-4537-99a1-f7154dc86a22-kube-api-access-g9lmd\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:18.410680 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.410639 4911 generic.go:356] "Generic (PLEG): container finished" podID="561bbf92-1f62-4537-99a1-f7154dc86a22" containerID="8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c" exitCode=0 Nov 01 16:47:18.410887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.410683 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" Nov 01 16:47:18.410887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.410725 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" event={"ID":"561bbf92-1f62-4537-99a1-f7154dc86a22","Type":"ContainerDied","Data":"8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c"} Nov 01 16:47:18.410887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.410767 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" event={"ID":"561bbf92-1f62-4537-99a1-f7154dc86a22","Type":"ContainerDied","Data":"81d19f85a28add0c987eab9e78ddd6252827790665da3ae500371a93ee4a7787"} Nov 01 16:47:18.410887 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.410785 4911 scope.go:122] "RemoveContainer" containerID="8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c" Nov 01 16:47:18.420536 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.420515 4911 scope.go:122] "RemoveContainer" containerID="8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c" Nov 01 16:47:18.420816 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:47:18.420797 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c\": container with ID starting with 8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c not found: ID does not exist" containerID="8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c" Nov 01 16:47:18.420866 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.420825 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c"} err="failed to get container status \"8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c\": rpc error: code = NotFound desc = could not find container \"8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c\": container with ID starting with 8593cbd3794b2d876b441db726df414745aa1fa291f668dd0e64b214f490c30c not found: ID does not exist" Nov 01 16:47:18.433987 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.433930 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc"] Nov 01 16:47:18.439032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:18.439005 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc"] Nov 01 16:47:19.133009 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:19.132967 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-230d0a/simulator-single-main-230d0a-kserve-9ff784f78-mgsdc" podUID="561bbf92-1f62-4537-99a1-f7154dc86a22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.202:8000/ready\": context deadline exceeded" Nov 01 16:47:19.964668 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:19.964622 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561bbf92-1f62-4537-99a1-f7154dc86a22" path="/var/lib/kubelet/pods/561bbf92-1f62-4537-99a1-f7154dc86a22/volumes" Nov 01 16:47:20.470030 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.469994 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h"] Nov 01 16:47:20.470530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.470410 4911 cpu_manager.go:414] "containerMap: removing container" podUID="1541d13a-91ee-4f67-8e36-4f520e011b15" containerName="revoke-keys" Nov 01 16:47:20.470530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.470423 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1541d13a-91ee-4f67-8e36-4f520e011b15" containerName="revoke-keys" Nov 01 16:47:20.470530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.470451 4911 cpu_manager.go:414] "containerMap: removing container" podUID="561bbf92-1f62-4537-99a1-f7154dc86a22" containerName="main" Nov 01 16:47:20.470530 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.470458 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561bbf92-1f62-4537-99a1-f7154dc86a22" containerName="main" Nov 01 16:47:20.470833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.470558 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="1541d13a-91ee-4f67-8e36-4f520e011b15" containerName="revoke-keys" Nov 01 16:47:20.470833 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.470575 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="561bbf92-1f62-4537-99a1-f7154dc86a22" containerName="main" Nov 01 16:47:20.474973 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.474951 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.483169 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.483118 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 16:47:20.484093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.484023 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 16:47:20.484435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.484411 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-93b14e1f-kserve-self-signed-certs\"" Nov 01 16:47:20.485556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.485434 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ffz56\"" Nov 01 16:47:20.485556 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.485437 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h"] Nov 01 16:47:20.510976 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.510933 4911 patch_prober.go:29] interesting pod/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:47:20.511162 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.510991 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:47:20.540270 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.540231 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tls-certs\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.540445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.540281 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfssd\" (UniqueName: \"kubernetes.io/projected/60dec7b4-b703-4e50-9ece-b0d8dff3e007-kube-api-access-gfssd\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.540445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.540353 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tmp-dir\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.540445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.540420 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-model-cache\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.540560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.540475 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-home\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.540560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.540540 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-dshm\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.641452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641411 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-dshm\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.641639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641475 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tls-certs\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.641639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641505 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfssd\" (UniqueName: \"kubernetes.io/projected/60dec7b4-b703-4e50-9ece-b0d8dff3e007-kube-api-access-gfssd\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.641639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641524 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tmp-dir\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.641639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641545 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-model-cache\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.641639 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641615 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-home\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.641982 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641952 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tmp-dir\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.642166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.641991 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-home\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.642166 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.642090 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-model-cache\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.643842 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.643815 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-dshm\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.644163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.644145 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tls-certs\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.653743 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.653715 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfssd\" (UniqueName: \"kubernetes.io/projected/60dec7b4-b703-4e50-9ece-b0d8dff3e007-kube-api-access-gfssd\") pod \"e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:20.792151 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:20.792113 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:21.070359 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.070326 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h"] Nov 01 16:47:21.072464 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:47:21.072436 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60dec7b4_b703_4e50_9ece_b0d8dff3e007.slice/crio-d6d7188a9886f3abcf14746455d574afe1b5115111b73a09a6aa6194b1a9d957 WatchSource:0}: Error finding container d6d7188a9886f3abcf14746455d574afe1b5115111b73a09a6aa6194b1a9d957: Status 404 returned error can't find the container with id d6d7188a9886f3abcf14746455d574afe1b5115111b73a09a6aa6194b1a9d957 Nov 01 16:47:21.079948 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.079929 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:47:21.146969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.146937 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-credential-socket\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147109 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147065 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e18849-e895-417d-8978-f9646f997c3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e18849-e895-417d-8978-f9646f997c3c-istiod-ca-cert\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147159 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147121 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-socket\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147206 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147179 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-envoy\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147194 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-credential-socket" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:21.147244 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147232 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-data\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147322 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147269 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-istio-token\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-istio-token\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147322 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147299 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5e18849-e895-417d-8978-f9646f997c3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e18849-e895-417d-8978-f9646f997c3c-istio-podinfo\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147335 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-kube-api-access-m4xks\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-kube-api-access-m4xks\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147409 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-certs\") pod \"d5e18849-e895-417d-8978-f9646f997c3c\" (UID: \"d5e18849-e895-417d-8978-f9646f997c3c\") " Nov 01 16:47:21.147524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147486 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-socket" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:21.147628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147527 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e18849-e895-417d-8978-f9646f997c3c-istiod-ca-cert" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:47:21.147755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147730 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-certs" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:21.147888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147814 4911 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.147888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147832 4911 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-credential-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.147888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147847 4911 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5e18849-e895-417d-8978-f9646f997c3c-istiod-ca-cert\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.147888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.147861 4911 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-workload-socket\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.148096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.148058 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-data" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:21.149709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.149683 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5e18849-e895-417d-8978-f9646f997c3c-istio-podinfo" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:47:21.149886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.149863 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-istio-token" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:21.149886 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.149870 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-envoy" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:47:21.151209 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.151189 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-kube-api-access-m4xks" pod "d5e18849-e895-417d-8978-f9646f997c3c" (UID: "d5e18849-e895-417d-8978-f9646f997c3c"). InnerVolumeSpecName "kube-api-access-m4xks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:21.249020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.248979 4911 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-envoy\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.249020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.249017 4911 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5e18849-e895-417d-8978-f9646f997c3c-istio-data\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.249284 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.249032 4911 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-istio-token\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.249284 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.249069 4911 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5e18849-e895-417d-8978-f9646f997c3c-istio-podinfo\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.249284 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.249084 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4xks\" (UniqueName: \"kubernetes.io/projected/d5e18849-e895-417d-8978-f9646f997c3c-kube-api-access-m4xks\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:21.430495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.430386 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" event={"ID":"60dec7b4-b703-4e50-9ece-b0d8dff3e007","Type":"ContainerStarted","Data":"66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281"} Nov 01 16:47:21.430495 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.430442 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" event={"ID":"60dec7b4-b703-4e50-9ece-b0d8dff3e007","Type":"ContainerStarted","Data":"d6d7188a9886f3abcf14746455d574afe1b5115111b73a09a6aa6194b1a9d957"} Nov 01 16:47:21.432313 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.432284 4911 generic.go:356] "Generic (PLEG): container finished" podID="d5e18849-e895-417d-8978-f9646f997c3c" containerID="48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e" exitCode=0 Nov 01 16:47:21.432477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.432337 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" Nov 01 16:47:21.432477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.432369 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" event={"ID":"d5e18849-e895-417d-8978-f9646f997c3c","Type":"ContainerDied","Data":"48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e"} Nov 01 16:47:21.432477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.432422 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8" event={"ID":"d5e18849-e895-417d-8978-f9646f997c3c","Type":"ContainerDied","Data":"0ad59862aaca04fbddc2417385916b5e0de9ba58e12c4985d56e7b5f0420247b"} Nov 01 16:47:21.432477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.432446 4911 scope.go:122] "RemoveContainer" containerID="48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e" Nov 01 16:47:21.442194 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.442176 4911 scope.go:122] "RemoveContainer" containerID="9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9" Nov 01 16:47:21.446904 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.446854 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" podStartSLOduration=1.446840874 podStartE2EDuration="1.446840874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:47:21.446414772 +0000 UTC m=+3581.590869413" watchObservedRunningTime="2026-11-01 16:47:21.446840874 +0000 UTC m=+3581.591295514" Nov 01 16:47:21.452708 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.452688 4911 scope.go:122] "RemoveContainer" containerID="48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e" Nov 01 16:47:21.453012 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:47:21.452982 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e\": container with ID starting with 48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e not found: ID does not exist" containerID="48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e" Nov 01 16:47:21.453138 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.453030 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e"} err="failed to get container status \"48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e\": rpc error: code = NotFound desc = could not find container \"48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e\": container with ID starting with 48d3f3cadb9389a1cc2359a1009431a5a1336dfde33ced41b311d96d9b405a7e not found: ID does not exist" Nov 01 16:47:21.453318 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.453074 4911 scope.go:122] "RemoveContainer" containerID="9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9" Nov 01 16:47:21.453670 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:47:21.453647 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9\": container with ID starting with 9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9 not found: ID does not exist" containerID="9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9" Nov 01 16:47:21.453751 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.453677 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9"} err="failed to get container status \"9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9\": rpc error: code = NotFound desc = could not find container \"9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9\": container with ID starting with 9e700b441a9cb42cd95c70b442ed7b052c306cb70f4fb2f3c0b333a45b193cd9 not found: ID does not exist" Nov 01 16:47:21.460550 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.460525 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8"] Nov 01 16:47:21.464277 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.464255 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-230d0a-openshift-default-55df6cdc97tx4f8"] Nov 01 16:47:21.792289 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.792244 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:21.973161 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:21.973122 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e18849-e895-417d-8978-f9646f997c3c" path="/var/lib/kubelet/pods/d5e18849-e895-417d-8978-f9646f997c3c/volumes" Nov 01 16:47:30.792596 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:30.792557 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:30.805070 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:30.805024 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:31.529020 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:31.528991 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:47:36.493417 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.493373 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx"] Nov 01 16:47:36.493829 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.493811 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" Nov 01 16:47:36.493875 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.493827 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" Nov 01 16:47:36.493929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.493918 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" Nov 01 16:47:36.501298 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.501266 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.510262 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.510227 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx"] Nov 01 16:47:36.510844 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.510812 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-93b14e1f-kserve-self-signed-certs\"" Nov 01 16:47:36.526685 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.526648 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-model-cache\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.526854 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.526787 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2bq\" (UniqueName: \"kubernetes.io/projected/e34cfd34-c78e-4852-8cc1-6f1a5652d315-kube-api-access-2g2bq\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.526916 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.526864 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-dshm\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.526916 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.526890 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tls-certs\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.527024 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.526914 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-home\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.527103 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.527084 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tmp-dir\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.627941 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.627891 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-model-cache\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.628303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.628277 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2bq\" (UniqueName: \"kubernetes.io/projected/e34cfd34-c78e-4852-8cc1-6f1a5652d315-kube-api-access-2g2bq\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.628803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.628777 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-dshm\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.628969 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.628951 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tls-certs\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.629133 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.629103 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-home\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.629257 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.629189 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tmp-dir\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.629475 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.628779 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-model-cache\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.629628 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.629543 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tmp-dir\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.629788 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.629754 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-home\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.632373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.632322 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tls-certs\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.632373 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.632363 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-dshm\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.637509 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.637482 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2bq\" (UniqueName: \"kubernetes.io/projected/e34cfd34-c78e-4852-8cc1-6f1a5652d315-kube-api-access-2g2bq\") pod \"e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:36.824211 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:36.824167 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:37.098477 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:37.098447 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx"] Nov 01 16:47:37.100320 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:47:37.100281 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode34cfd34_c78e_4852_8cc1_6f1a5652d315.slice/crio-39aac19d29dddc2a7da804d67f8c8d47ac0e9940abd36509c2a3b361bf31b5df WatchSource:0}: Error finding container 39aac19d29dddc2a7da804d67f8c8d47ac0e9940abd36509c2a3b361bf31b5df: Status 404 returned error can't find the container with id 39aac19d29dddc2a7da804d67f8c8d47ac0e9940abd36509c2a3b361bf31b5df Nov 01 16:47:37.553083 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:37.553027 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" event={"ID":"e34cfd34-c78e-4852-8cc1-6f1a5652d315","Type":"ContainerStarted","Data":"1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1"} Nov 01 16:47:37.553488 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:37.553093 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" event={"ID":"e34cfd34-c78e-4852-8cc1-6f1a5652d315","Type":"ContainerStarted","Data":"39aac19d29dddc2a7da804d67f8c8d47ac0e9940abd36509c2a3b361bf31b5df"} Nov 01 16:47:37.572769 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:37.572718 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" podStartSLOduration=1.57270538 podStartE2EDuration="1.57270538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:47:37.569506076 +0000 UTC m=+3597.713960717" watchObservedRunningTime="2026-11-01 16:47:37.57270538 +0000 UTC m=+3597.717160021" Nov 01 16:47:37.825361 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:37.825276 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:40.938626 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:47:40.938587 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:47:40.963063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:40.962954 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:47:40.970946 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:40.970759 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:47:46.825227 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:46.825191 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:46.837968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:46.837939 4911 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:47:47.651661 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:47:47.651624 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:48:04.286413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:04.286370 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h"] Nov 01 16:48:04.925110 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:04.925067 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx"] Nov 01 16:48:10.941230 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:48:10.941182 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:48:14.019803 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:14.019766 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk"] Nov 01 16:48:14.022599 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:14.022566 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d7b577-6llwk"] Nov 01 16:48:15.964974 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:15.964936 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e" path="/var/lib/kubelet/pods/7cdd046a-6ea5-4ca4-87e7-7cf40a68a54e/volumes" Nov 01 16:48:19.318888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.318837 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" podUID="60dec7b4-b703-4e50-9ece-b0d8dff3e007" containerName="main" containerID="cri-o://66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281" gracePeriod=45 Nov 01 16:48:19.617308 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.617283 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:48:19.735118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735074 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tmp-dir\") pod \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " Nov 01 16:48:19.735327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735128 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-model-cache\") pod \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " Nov 01 16:48:19.735327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735191 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tls-certs\" (UniqueName: \"kubernetes.io/secret/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tls-certs\") pod \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " Nov 01 16:48:19.735327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735215 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-home\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-home\") pod \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " Nov 01 16:48:19.735327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735269 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-dshm\") pod \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " Nov 01 16:48:19.735327 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735302 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60dec7b4-b703-4e50-9ece-b0d8dff3e007-kube-api-access-gfssd\" (UniqueName: \"kubernetes.io/projected/60dec7b4-b703-4e50-9ece-b0d8dff3e007-kube-api-access-gfssd\") pod \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\" (UID: \"60dec7b4-b703-4e50-9ece-b0d8dff3e007\") " Nov 01 16:48:19.735603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735377 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tmp-dir" pod "60dec7b4-b703-4e50-9ece-b0d8dff3e007" (UID: "60dec7b4-b703-4e50-9ece-b0d8dff3e007"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:19.735603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735435 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-model-cache" pod "60dec7b4-b703-4e50-9ece-b0d8dff3e007" (UID: "60dec7b4-b703-4e50-9ece-b0d8dff3e007"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:19.735603 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735548 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-home" pod "60dec7b4-b703-4e50-9ece-b0d8dff3e007" (UID: "60dec7b4-b703-4e50-9ece-b0d8dff3e007"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:19.735724 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735702 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:19.735762 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735732 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:19.735762 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.735748 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:19.737608 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.737579 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-dshm" pod "60dec7b4-b703-4e50-9ece-b0d8dff3e007" (UID: "60dec7b4-b703-4e50-9ece-b0d8dff3e007"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:19.737709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.737663 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tls-certs" pod "60dec7b4-b703-4e50-9ece-b0d8dff3e007" (UID: "60dec7b4-b703-4e50-9ece-b0d8dff3e007"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:19.737776 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.737752 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dec7b4-b703-4e50-9ece-b0d8dff3e007-kube-api-access-gfssd" pod "60dec7b4-b703-4e50-9ece-b0d8dff3e007" (UID: "60dec7b4-b703-4e50-9ece-b0d8dff3e007"). InnerVolumeSpecName "kube-api-access-gfssd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:48:19.836452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.836414 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60dec7b4-b703-4e50-9ece-b0d8dff3e007-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:19.836452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.836449 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfssd\" (UniqueName: \"kubernetes.io/projected/60dec7b4-b703-4e50-9ece-b0d8dff3e007-kube-api-access-gfssd\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:19.836452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.836460 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60dec7b4-b703-4e50-9ece-b0d8dff3e007-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:19.942263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.942170 4911 generic.go:356] "Generic (PLEG): container finished" podID="60dec7b4-b703-4e50-9ece-b0d8dff3e007" containerID="66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281" exitCode=0 Nov 01 16:48:19.942263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.942212 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" Nov 01 16:48:19.942263 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.942252 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" event={"ID":"60dec7b4-b703-4e50-9ece-b0d8dff3e007","Type":"ContainerDied","Data":"66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281"} Nov 01 16:48:19.942521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.942298 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" event={"ID":"60dec7b4-b703-4e50-9ece-b0d8dff3e007","Type":"ContainerDied","Data":"d6d7188a9886f3abcf14746455d574afe1b5115111b73a09a6aa6194b1a9d957"} Nov 01 16:48:19.942521 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.942320 4911 scope.go:122] "RemoveContainer" containerID="66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281" Nov 01 16:48:19.948718 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.948681 4911 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" podUID="e34cfd34-c78e-4852-8cc1-6f1a5652d315" containerName="main" containerID="cri-o://1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1" gracePeriod=45 Nov 01 16:48:19.952824 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.952800 4911 scope.go:122] "RemoveContainer" containerID="66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281" Nov 01 16:48:19.953133 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:48:19.953109 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281\": container with ID starting with 66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281 not found: ID does not exist" containerID="66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281" Nov 01 16:48:19.953198 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.953145 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281"} err="failed to get container status \"66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281\": rpc error: code = NotFound desc = could not find container \"66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281\": container with ID starting with 66c9f940769b51d76d330f0faf721d2c2ad438b5caa1f47cdf1a4e82fa2db281 not found: ID does not exist" Nov 01 16:48:19.967589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.967561 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h"] Nov 01 16:48:19.972920 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:19.972887 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h"] Nov 01 16:48:20.190653 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.190626 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341439 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tmp-dir\") pod \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341522 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tls-certs\" (UniqueName: \"kubernetes.io/secret/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tls-certs\") pod \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341590 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-home\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-home\") pod \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341623 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-model-cache\") pod \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341665 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34cfd34-c78e-4852-8cc1-6f1a5652d315-kube-api-access-2g2bq\" (UniqueName: \"kubernetes.io/projected/e34cfd34-c78e-4852-8cc1-6f1a5652d315-kube-api-access-2g2bq\") pod \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341692 4911 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-dshm\") pod \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\" (UID: \"e34cfd34-c78e-4852-8cc1-6f1a5652d315\") " Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341764 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tmp-dir" pod "e34cfd34-c78e-4852-8cc1-6f1a5652d315" (UID: "e34cfd34-c78e-4852-8cc1-6f1a5652d315"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.341981 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-model-cache" pod "e34cfd34-c78e-4852-8cc1-6f1a5652d315" (UID: "e34cfd34-c78e-4852-8cc1-6f1a5652d315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.342071 4911 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tmp-dir\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:20.342105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.342079 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-home" pod "e34cfd34-c78e-4852-8cc1-6f1a5652d315" (UID: "e34cfd34-c78e-4852-8cc1-6f1a5652d315"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:20.344413 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.344382 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34cfd34-c78e-4852-8cc1-6f1a5652d315-kube-api-access-2g2bq" pod "e34cfd34-c78e-4852-8cc1-6f1a5652d315" (UID: "e34cfd34-c78e-4852-8cc1-6f1a5652d315"). InnerVolumeSpecName "kube-api-access-2g2bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:48:20.344551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.344478 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tls-certs" pod "e34cfd34-c78e-4852-8cc1-6f1a5652d315" (UID: "e34cfd34-c78e-4852-8cc1-6f1a5652d315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:20.345236 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.345214 4911 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-dshm" pod "e34cfd34-c78e-4852-8cc1-6f1a5652d315" (UID: "e34cfd34-c78e-4852-8cc1-6f1a5652d315"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:48:20.442929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.442884 4911 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34cfd34-c78e-4852-8cc1-6f1a5652d315-tls-certs\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:20.442929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.442920 4911 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-home\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:20.442929 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.442934 4911 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-model-cache\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:20.443197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.442949 4911 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2bq\" (UniqueName: \"kubernetes.io/projected/e34cfd34-c78e-4852-8cc1-6f1a5652d315-kube-api-access-2g2bq\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:20.443197 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.442962 4911 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34cfd34-c78e-4852-8cc1-6f1a5652d315-dshm\") on node \"ip-10-0-143-101.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:20.519403 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.519353 4911 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-93b14e1f-kserve-7d7d8db849-q774h" podUID="60dec7b4-b703-4e50-9ece-b0d8dff3e007" containerName="main" probeResult="failure" output="Get \"https://10.132.0.205:8000/ready\": context deadline exceeded" Nov 01 16:48:20.951693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.951596 4911 generic.go:356] "Generic (PLEG): container finished" podID="e34cfd34-c78e-4852-8cc1-6f1a5652d315" containerID="1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1" exitCode=0 Nov 01 16:48:20.951693 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.951641 4911 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" Nov 01 16:48:20.951950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.951686 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" event={"ID":"e34cfd34-c78e-4852-8cc1-6f1a5652d315","Type":"ContainerDied","Data":"1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1"} Nov 01 16:48:20.951950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.951866 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx" event={"ID":"e34cfd34-c78e-4852-8cc1-6f1a5652d315","Type":"ContainerDied","Data":"39aac19d29dddc2a7da804d67f8c8d47ac0e9940abd36509c2a3b361bf31b5df"} Nov 01 16:48:20.951950 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.951895 4911 scope.go:122] "RemoveContainer" containerID="1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1" Nov 01 16:48:20.965937 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.965913 4911 scope.go:122] "RemoveContainer" containerID="1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1" Nov 01 16:48:20.966314 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:48:20.966289 4911 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1\": container with ID starting with 1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1 not found: ID does not exist" containerID="1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1" Nov 01 16:48:20.966445 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.966320 4911 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1"} err="failed to get container status \"1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1\": rpc error: code = NotFound desc = could not find container \"1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1\": container with ID starting with 1d3fea750d25504c68da185684e35e437fc1c7b76ab2d3a6394a64d5ba6521e1 not found: ID does not exist" Nov 01 16:48:20.977438 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.977409 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx"] Nov 01 16:48:20.979357 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:20.979330 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-93b14e1f-kserve-7676bd6db5-8f8sx"] Nov 01 16:48:22.028424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:22.028379 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60dec7b4-b703-4e50-9ece-b0d8dff3e007" path="/var/lib/kubelet/pods/60dec7b4-b703-4e50-9ece-b0d8dff3e007/volumes" Nov 01 16:48:22.029392 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:22.029369 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34cfd34-c78e-4852-8cc1-6f1a5652d315" path="/var/lib/kubelet/pods/e34cfd34-c78e-4852-8cc1-6f1a5652d315/volumes" Nov 01 16:48:38.962919 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.962868 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796985b8cc-9mq78"] Nov 01 16:48:38.963651 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963626 4911 cpu_manager.go:414] "containerMap: removing container" podUID="60dec7b4-b703-4e50-9ece-b0d8dff3e007" containerName="main" Nov 01 16:48:38.963709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963654 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dec7b4-b703-4e50-9ece-b0d8dff3e007" containerName="main" Nov 01 16:48:38.963709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963669 4911 cpu_manager.go:414] "containerMap: removing container" podUID="e34cfd34-c78e-4852-8cc1-6f1a5652d315" containerName="main" Nov 01 16:48:38.963709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963677 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34cfd34-c78e-4852-8cc1-6f1a5652d315" containerName="main" Nov 01 16:48:38.963709 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963698 4911 cpu_manager.go:414] "containerMap: removing container" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" Nov 01 16:48:38.963841 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963709 4911 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" Nov 01 16:48:38.963876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963850 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e18849-e895-417d-8978-f9646f997c3c" containerName="istio-proxy" Nov 01 16:48:38.963876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963870 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dec7b4-b703-4e50-9ece-b0d8dff3e007" containerName="main" Nov 01 16:48:38.963943 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.963880 4911 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34cfd34-c78e-4852-8cc1-6f1a5652d315" containerName="main" Nov 01 16:48:38.969279 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.969252 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:38.979096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.978346 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 16:48:38.979096 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.978412 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796985b8cc-9mq78"] Nov 01 16:48:38.979607 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:38.979567 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 16:48:39.049808 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.049770 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 16:48:39.049968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.049816 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 16:48:39.049968 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.049867 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 16:48:39.056142 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.056123 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1793129-90ec-4356-a749-d0515d351249-openshift-service-ca\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.056269 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.056252 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1793129-90ec-4356-a749-d0515d351249-metrics-certs\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.056384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.056360 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1793129-90ec-4356-a749-d0515d351249-maas-api-tls\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.056436 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.056418 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6w6x\" (UniqueName: \"kubernetes.io/projected/d1793129-90ec-4356-a749-d0515d351249-kube-api-access-q6w6x\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.157251 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.157211 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1793129-90ec-4356-a749-d0515d351249-openshift-service-ca\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.157435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.157308 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1793129-90ec-4356-a749-d0515d351249-metrics-certs\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.157435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.157373 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1793129-90ec-4356-a749-d0515d351249-maas-api-tls\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.157435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.157413 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6w6x\" (UniqueName: \"kubernetes.io/projected/d1793129-90ec-4356-a749-d0515d351249-kube-api-access-q6w6x\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.158026 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.158001 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1793129-90ec-4356-a749-d0515d351249-openshift-service-ca\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.159831 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.159802 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1793129-90ec-4356-a749-d0515d351249-metrics-certs\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.159942 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.159864 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1793129-90ec-4356-a749-d0515d351249-maas-api-tls\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.165372 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.165350 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6w6x\" (UniqueName: \"kubernetes.io/projected/d1793129-90ec-4356-a749-d0515d351249-kube-api-access-q6w6x\") pod \"maas-api-796985b8cc-9mq78\" (UID: \"d1793129-90ec-4356-a749-d0515d351249\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.289876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.289844 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:39.563978 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.563951 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796985b8cc-9mq78"] Nov 01 16:48:39.565842 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:48:39.565812 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1793129_90ec_4356_a749_d0515d351249.slice/crio-17298090d0ea6f7e1b8336adcbd9b558241a8dcac78359682dc035242d342749 WatchSource:0}: Error finding container 17298090d0ea6f7e1b8336adcbd9b558241a8dcac78359682dc035242d342749: Status 404 returned error can't find the container with id 17298090d0ea6f7e1b8336adcbd9b558241a8dcac78359682dc035242d342749 Nov 01 16:48:39.567126 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:39.567106 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:48:40.133235 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:40.133205 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" event={"ID":"d1793129-90ec-4356-a749-d0515d351249","Type":"ContainerStarted","Data":"17298090d0ea6f7e1b8336adcbd9b558241a8dcac78359682dc035242d342749"} Nov 01 16:48:40.866508 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:48:40.866469 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:48:41.140614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:41.140518 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" event={"ID":"d1793129-90ec-4356-a749-d0515d351249","Type":"ContainerStarted","Data":"422d173524ffbfb507f3a7d63601edb102209c36f0fdd3e8d6b3b34e11b3d0b0"} Nov 01 16:48:41.140614 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:41.140589 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:48:41.159143 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:41.159095 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" podStartSLOduration=2.6141966610000003 podStartE2EDuration="3.159079418s" totalImagesPullingTime="544.882757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:41.158462435 +0000 UTC m=+3661.302917076" watchObservedRunningTime="2026-11-01 16:48:41.159079418 +0000 UTC m=+3661.303534064" Nov 01 16:48:47.151987 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:48:47.151957 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796985b8cc-9mq78" Nov 01 16:49:10.865743 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:49:10.865702 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:49:40.866646 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:49:40.866605 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:49:43.006071 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:49:43.003963 4911 scope.go:122] "RemoveContainer" containerID="70fcb3ffdf9f96e3ddf5ade0bb204c7bff96b9c0d6ba6dab406247db1e61e58d" Nov 01 16:49:43.023597 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:49:43.023570 4911 scope.go:122] "RemoveContainer" containerID="b8a686530aa9986bcff672c1ce715b731e0c96d7824e5acca2a252dc84f1de87" Nov 01 16:50:10.866152 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:50:10.866108 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:50:40.869706 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:50:40.869665 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:51:10.866147 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:51:10.866109 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:51:40.864469 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:51:40.864429 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:51:51.037634 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:51:51.037596 4911 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj"] Nov 01 16:51:51.045422 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:51:51.045392 4911 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33384c4a-6ksgj"] Nov 01 16:51:51.964695 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:51:51.964657 4911 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da37491-0107-4491-a065-77c2c865eebf" path="/var/lib/kubelet/pods/7da37491-0107-4491-a065-77c2c865eebf/volumes" Nov 01 16:52:09.050118 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:09.050074 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5446d75d-76bmb_233ad863-8cfe-4c57-9250-f07bb5329feb/authorino/0.log" Nov 01 16:52:10.881476 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:52:10.881423 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:52:19.771755 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:19.771722 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dqb7v_6d3f317c-4ee8-4892-bbbb-ee45450abcb6/manager/0.log" Nov 01 16:52:20.176309 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:20.176223 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sbw8k_62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4/manager/0.log" Nov 01 16:52:20.654175 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:20.654144 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-tqcrs_9dae4e15-1cb1-4c30-b15a-25b32cbd939b/server/0.log" Nov 01 16:52:21.035239 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:21.035207 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hzdp2_07a610d5-7012-466d-9d51-ff68f7119b52/proxy/0.log" Nov 01 16:52:21.755334 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:21.755276 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796985b8cc-9mq78_d1793129-90ec-4356-a749-d0515d351249/maas-api/0.log" Nov 01 16:52:24.715551 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:24.715509 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5446d75d-76bmb_233ad863-8cfe-4c57-9250-f07bb5329feb/authorino/0.log" Nov 01 16:52:24.947292 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:24.947256 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8m4nz_c11ba1b9-a4fa-45ea-adbc-b13aea773ca0/manager/0.log" Nov 01 16:52:25.136303 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:25.136272 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v26kw_59d8fcb3-1620-414d-a8e4-104c9d63e26d/manager/0.log" Nov 01 16:52:25.925087 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:25.925034 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4ntsq_3f3b6fab-75bb-416f-a042-125cf512aa4e/manager/0.log" Nov 01 16:52:26.662306 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:26.662253 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k_1fa9f79a-f263-46fd-8556-5fc1027e3b61/istio-proxy/0.log" Nov 01 16:52:27.754485 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:27.754446 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-9rb5l_f37e3bdc-1a02-41cb-8c76-61d11cbd86b8/istio-proxy/1.log" Nov 01 16:52:28.706435 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:28.706399 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-s62zq_db534cd4-9441-4dab-abe4-f8109f12f529/istio-proxy/0.log" Nov 01 16:52:29.048176 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:29.048143 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vhjzm_a0181ade-77fb-4c16-adda-22dca4ce2524/istio-proxy/0.log" Nov 01 16:52:29.229675 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:29.229641 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll_d302b57f-5184-48fd-b8e3-0e22166dd567/istio-proxy/0.log" Nov 01 16:52:29.791032 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:29.791000 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69f6694997-vvcm2_e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/router/0.log" Nov 01 16:52:30.231906 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:30.231826 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-9rb5l_f37e3bdc-1a02-41cb-8c76-61d11cbd86b8/istio-proxy/0.log" Nov 01 16:52:32.885723 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:32.885683 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5446d75d-76bmb_233ad863-8cfe-4c57-9250-f07bb5329feb/authorino/0.log" Nov 01 16:52:40.865753 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:52:40.865712 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:52:41.018615 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:41.018585 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:52:41.028856 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:41.028832 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:52:41.890207 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:41.890174 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dqb7v_6d3f317c-4ee8-4892-bbbb-ee45450abcb6/manager/0.log" Nov 01 16:52:42.270888 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:42.270854 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sbw8k_62ad4809-9d4b-4a4c-8ea2-26cf5cb427b4/manager/0.log" Nov 01 16:52:42.676681 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:42.676596 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-tqcrs_9dae4e15-1cb1-4c30-b15a-25b32cbd939b/server/0.log" Nov 01 16:52:43.080293 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:43.080264 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hzdp2_07a610d5-7012-466d-9d51-ff68f7119b52/proxy/0.log" Nov 01 16:52:43.589424 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:43.589390 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796985b8cc-9mq78_d1793129-90ec-4356-a749-d0515d351249/maas-api/0.log" Nov 01 16:52:46.044555 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:46.044523 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5446d75d-76bmb_233ad863-8cfe-4c57-9250-f07bb5329feb/authorino/0.log" Nov 01 16:52:46.247908 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:46.247877 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8m4nz_c11ba1b9-a4fa-45ea-adbc-b13aea773ca0/manager/0.log" Nov 01 16:52:46.436960 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:46.436881 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v26kw_59d8fcb3-1620-414d-a8e4-104c9d63e26d/manager/0.log" Nov 01 16:52:47.201062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:47.201012 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4ntsq_3f3b6fab-75bb-416f-a042-125cf512aa4e/manager/0.log" Nov 01 16:52:47.742766 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:47.742733 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k_1fa9f79a-f263-46fd-8556-5fc1027e3b61/istio-proxy/0.log" Nov 01 16:52:48.828560 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:48.828516 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-9rb5l_f37e3bdc-1a02-41cb-8c76-61d11cbd86b8/istio-proxy/1.log" Nov 01 16:52:49.721465 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:49.721431 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-s62zq_db534cd4-9441-4dab-abe4-f8109f12f529/istio-proxy/0.log" Nov 01 16:52:50.085701 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:50.085666 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vhjzm_a0181ade-77fb-4c16-adda-22dca4ce2524/istio-proxy/0.log" Nov 01 16:52:50.274163 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:50.274129 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll_d302b57f-5184-48fd-b8e3-0e22166dd567/istio-proxy/0.log" Nov 01 16:52:50.805112 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:52:50.805078 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69f6694997-vvcm2_e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/router/0.log" Nov 01 16:53:10.865094 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:53:10.865027 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:53:40.865520 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:53:40.865480 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:53:43.079454 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:53:43.079421 4911 scope.go:122] "RemoveContainer" containerID="94dc98ac976f1e75ec9e00ab7af20e5c6ab2e4826a21e8815d34f7edb7315ad2" Nov 01 16:53:43.088665 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:53:43.088636 4911 scope.go:122] "RemoveContainer" containerID="93e48286d0c08529d0cda7cce973642d34e895f38de6a06c9bbf04ec9ee09d62" Nov 01 16:54:10.865710 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:54:10.865668 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:54:18.582542 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.582505 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c79r4/must-gather-jbvqq"] Nov 01 16:54:18.586213 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.586193 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:18.591731 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.591707 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c79r4\"/\"openshift-service-ca.crt\"" Nov 01 16:54:18.592809 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.592778 4911 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c79r4\"/\"kube-root-ca.crt\"" Nov 01 16:54:18.592961 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.592821 4911 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c79r4\"/\"default-dockercfg-fqmbm\"" Nov 01 16:54:18.599398 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.599373 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/must-gather-jbvqq"] Nov 01 16:54:18.643735 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.643694 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/31fdcad1-25c5-4188-856f-472f8a81335e-kube-api-access-gb477\") pod \"must-gather-jbvqq\" (UID: \"31fdcad1-25c5-4188-856f-472f8a81335e\") " pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:18.643914 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.643809 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31fdcad1-25c5-4188-856f-472f8a81335e-must-gather-output\") pod \"must-gather-jbvqq\" (UID: \"31fdcad1-25c5-4188-856f-472f8a81335e\") " pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:18.744355 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.744315 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/31fdcad1-25c5-4188-856f-472f8a81335e-kube-api-access-gb477\") pod \"must-gather-jbvqq\" (UID: \"31fdcad1-25c5-4188-856f-472f8a81335e\") " pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:18.744570 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.744465 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31fdcad1-25c5-4188-856f-472f8a81335e-must-gather-output\") pod \"must-gather-jbvqq\" (UID: \"31fdcad1-25c5-4188-856f-472f8a81335e\") " pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:18.744807 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.744785 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/31fdcad1-25c5-4188-856f-472f8a81335e-must-gather-output\") pod \"must-gather-jbvqq\" (UID: \"31fdcad1-25c5-4188-856f-472f8a81335e\") " pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:18.755185 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.755160 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb477\" (UniqueName: \"kubernetes.io/projected/31fdcad1-25c5-4188-856f-472f8a81335e-kube-api-access-gb477\") pod \"must-gather-jbvqq\" (UID: \"31fdcad1-25c5-4188-856f-472f8a81335e\") " pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:18.901662 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:18.901576 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/must-gather-jbvqq" Nov 01 16:54:19.180105 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:19.180072 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/must-gather-jbvqq"] Nov 01 16:54:19.181914 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:54:19.181886 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31fdcad1_25c5_4188_856f_472f8a81335e.slice/crio-c9d810825ff7e2beaa377282bfa5bfa1b8556a5503b7fadea43897386f552fbe WatchSource:0}: Error finding container c9d810825ff7e2beaa377282bfa5bfa1b8556a5503b7fadea43897386f552fbe: Status 404 returned error can't find the container with id c9d810825ff7e2beaa377282bfa5bfa1b8556a5503b7fadea43897386f552fbe Nov 01 16:54:19.183736 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:19.183715 4911 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:54:20.020510 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:20.020475 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/must-gather-jbvqq" event={"ID":"31fdcad1-25c5-4188-856f-472f8a81335e","Type":"ContainerStarted","Data":"c9d810825ff7e2beaa377282bfa5bfa1b8556a5503b7fadea43897386f552fbe"} Nov 01 16:54:21.033093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:21.032991 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/must-gather-jbvqq" event={"ID":"31fdcad1-25c5-4188-856f-472f8a81335e","Type":"ContainerStarted","Data":"ea9e76e485ba47909e6cb7a83b48240c7385c5f20cf0d276ff0de4f382930510"} Nov 01 16:54:21.033093 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:21.033057 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/must-gather-jbvqq" event={"ID":"31fdcad1-25c5-4188-856f-472f8a81335e","Type":"ContainerStarted","Data":"baa6def0063675c00b71dda32b9687186835d31b959ca274d1051db6a99f31ae"} Nov 01 16:54:21.057926 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:21.057864 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c79r4/must-gather-jbvqq" podStartSLOduration=2.184998119 podStartE2EDuration="3.057849293s" totalImagesPullingTime="872.851174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:54:21.055211095 +0000 UTC m=+4001.199665737" watchObservedRunningTime="2026-11-01 16:54:21.057849293 +0000 UTC m=+4001.202303928" Nov 01 16:54:22.243092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:22.242803 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xvq6f_e9337a97-e8fa-4061-ba5f-fea449c7d390/global-pull-secret-syncer/0.log" Nov 01 16:54:22.282688 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:22.282655 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5qrs6_3be78730-9048-43e9-9d31-e3819027d095/connection-checker/0.log" Nov 01 16:54:22.310876 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:22.310847 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6v9fs_1a7cd5bd-620b-4d98-b64a-4582788161a4/connection-checker/0.log" Nov 01 16:54:22.338774 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:22.338747 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-g4sxx_85d4d845-68cc-44e8-9503-5969ed5fcc1b/connection-checker/0.log" Nov 01 16:54:22.526441 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:22.526355 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wvndh_e01f94e8-cf07-45d8-b023-0f87e7d41cdf/konnectivity-agent/0.log" Nov 01 16:54:22.627261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:22.627227 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-101.us-east-2.compute.internal_fd6f673b54e13544e2f8008a04d167a7/haproxy/0.log" Nov 01 16:54:31.221642 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:31.221603 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5446d75d-76bmb_233ad863-8cfe-4c57-9250-f07bb5329feb/authorino/0.log" Nov 01 16:54:31.261018 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:31.260984 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8m4nz_c11ba1b9-a4fa-45ea-adbc-b13aea773ca0/manager/0.log" Nov 01 16:54:31.288798 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:31.288750 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-v26kw_59d8fcb3-1620-414d-a8e4-104c9d63e26d/manager/0.log" Nov 01 16:54:31.564062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:31.563381 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4ntsq_3f3b6fab-75bb-416f-a042-125cf512aa4e/manager/0.log" Nov 01 16:54:32.673539 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.673505 4911 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq"] Nov 01 16:54:32.684402 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.684366 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.687383 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.687343 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq"] Nov 01 16:54:32.774976 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.774937 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-59xc9_57440ce3-722e-4835-927a-e8f2fac8b3c7/cluster-monitoring-operator/0.log" Nov 01 16:54:32.820806 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.820770 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-lib-modules\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.820983 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.820934 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-sys\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.821072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.820981 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-proc\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.821072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.821030 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq25s\" (UniqueName: \"kubernetes.io/projected/dd01e26f-6422-402d-9931-79dca596f7f0-kube-api-access-kq25s\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.821196 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.821175 4911 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-podres\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922390 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922341 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-sys\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922404 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-proc\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922440 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq25s\" (UniqueName: \"kubernetes.io/projected/dd01e26f-6422-402d-9931-79dca596f7f0-kube-api-access-kq25s\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922479 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-sys\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922487 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-podres\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922555 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-podres\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922566 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-proc\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922612 4911 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-lib-modules\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.922845 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.922715 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd01e26f-6422-402d-9931-79dca596f7f0-lib-modules\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:32.932442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:32.932356 4911 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq25s\" (UniqueName: \"kubernetes.io/projected/dd01e26f-6422-402d-9931-79dca596f7f0-kube-api-access-kq25s\") pod \"perf-node-gather-daemonset-fcgnq\" (UID: \"dd01e26f-6422-402d-9931-79dca596f7f0\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:33.012164 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:33.011206 4911 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:33.126410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:33.126382 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wgx5w_ca68b7f9-2b1d-4b17-81e6-68c64ce8428d/node-exporter/0.log" Nov 01 16:54:33.151023 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:33.150991 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wgx5w_ca68b7f9-2b1d-4b17-81e6-68c64ce8428d/kube-rbac-proxy/0.log" Nov 01 16:54:33.175825 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:33.175797 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wgx5w_ca68b7f9-2b1d-4b17-81e6-68c64ce8428d/init-textfile/0.log" Nov 01 16:54:33.301076 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:33.299006 4911 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq"] Nov 01 16:54:33.307087 ip-10-0-143-101 kubenswrapper[4911]: W1101 16:54:33.307022 4911 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd01e26f_6422_402d_9931_79dca596f7f0.slice/crio-e4d471d7ab43c1123dfe9528a197f9be8c83d25c45a252ba37f95533d25fa4f5 WatchSource:0}: Error finding container e4d471d7ab43c1123dfe9528a197f9be8c83d25c45a252ba37f95533d25fa4f5: Status 404 returned error can't find the container with id e4d471d7ab43c1123dfe9528a197f9be8c83d25c45a252ba37f95533d25fa4f5 Nov 01 16:54:34.207569 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:34.207515 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" event={"ID":"dd01e26f-6422-402d-9931-79dca596f7f0","Type":"ContainerStarted","Data":"cc225a3d524fd8f01e84205cad595c868f4ac68b9799f25f92a25e7d056fb6e0"} Nov 01 16:54:34.208008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:34.207605 4911 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" event={"ID":"dd01e26f-6422-402d-9931-79dca596f7f0","Type":"ContainerStarted","Data":"e4d471d7ab43c1123dfe9528a197f9be8c83d25c45a252ba37f95533d25fa4f5"} Nov 01 16:54:34.208008 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:34.207663 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:34.229062 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:34.228122 4911 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" podStartSLOduration=2.228103871 podStartE2EDuration="2.228103871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 16:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:54:34.225442994 +0000 UTC m=+4014.369897636" watchObservedRunningTime="2026-11-01 16:54:34.228103871 +0000 UTC m=+4014.372558514" Nov 01 16:54:34.890380 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:34.890341 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-j46fh_19f38b5c-f14d-48fd-b0cf-a2581c62384a/networking-console-plugin/0.log" Nov 01 16:54:35.513307 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:35.513274 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/3.log" Nov 01 16:54:35.521001 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:35.520970 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qhqp_cea9e9fb-3362-48cf-b2ec-3db89f981464/console-operator/4.log" Nov 01 16:54:36.830707 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:36.830674 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-xs4tv_f1f800e1-24b3-4323-97f3-2009efb63d1b/volume-data-source-validator/0.log" Nov 01 16:54:38.163582 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:38.163542 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jx495_0e085d33-5390-4d2e-a616-61160ee87e46/dns/0.log" Nov 01 16:54:38.187128 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:38.187095 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jx495_0e085d33-5390-4d2e-a616-61160ee87e46/kube-rbac-proxy/0.log" Nov 01 16:54:38.271089 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:38.271025 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wx7cw_d72732eb-0380-4f65-ab26-a788f64ad3fd/dns-node-resolver/0.log" Nov 01 16:54:38.954524 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:38.954487 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-czbft_f237b168-098b-4df7-a56e-f245691a8086/node-ca/0.log" Nov 01 16:54:40.142410 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:40.142358 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k_1fa9f79a-f263-46fd-8556-5fc1027e3b61/istio-proxy/0.log" Nov 01 16:54:40.224797 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:40.224772 4911 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-fcgnq" Nov 01 16:54:40.874983 ip-10-0-143-101 kubenswrapper[4911]: E1101 16:54:40.874941 4911 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6x8kz" containerName="extractor" Nov 01 16:54:43.120261 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:43.120226 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f76mh9k_1fa9f79a-f263-46fd-8556-5fc1027e3b61/istio-proxy/0.log" Nov 01 16:54:43.821678 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:43.821646 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-9rb5l_f37e3bdc-1a02-41cb-8c76-61d11cbd86b8/istio-proxy/0.log" Nov 01 16:54:43.823792 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:43.823757 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-9rb5l_f37e3bdc-1a02-41cb-8c76-61d11cbd86b8/istio-proxy/1.log" Nov 01 16:54:44.008792 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:44.008754 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-s62zq_db534cd4-9441-4dab-abe4-f8109f12f529/istio-proxy/0.log" Nov 01 16:54:44.104466 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:44.104384 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vhjzm_a0181ade-77fb-4c16-adda-22dca4ce2524/istio-proxy/0.log" Nov 01 16:54:44.138384 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:44.138353 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll_d302b57f-5184-48fd-b8e3-0e22166dd567/istio-proxy/0.log" Nov 01 16:54:44.262063 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:44.258127 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69f6694997-vvcm2_e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/router/0.log" Nov 01 16:54:46.424072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:46.420519 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-9rb5l_f37e3bdc-1a02-41cb-8c76-61d11cbd86b8/istio-proxy/0.log" Nov 01 16:54:46.424072 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:46.422028 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-9rb5l_f37e3bdc-1a02-41cb-8c76-61d11cbd86b8/istio-proxy/1.log" Nov 01 16:54:46.517310 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:46.517273 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-s62zq_db534cd4-9441-4dab-abe4-f8109f12f529/istio-proxy/0.log" Nov 01 16:54:46.555931 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:46.555887 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-vhjzm_a0181ade-77fb-4c16-adda-22dca4ce2524/istio-proxy/0.log" Nov 01 16:54:46.572223 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:46.572176 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-shrll_d302b57f-5184-48fd-b8e3-0e22166dd567/istio-proxy/0.log" Nov 01 16:54:46.599779 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:46.599740 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cf2mk_8f6d982a-e0d9-42ee-b342-290a02120b36/serve-healthcheck-canary/0.log" Nov 01 16:54:46.632092 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:46.632060 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69f6694997-vvcm2_e3d66b38-3dfa-46eb-a2c7-d8a5de28be89/router/0.log" Nov 01 16:54:47.210589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:47.207959 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fz7xr_23a31617-3eb9-499e-a192-f4d24a34e835/insights-operator/0.log" Nov 01 16:54:47.210589 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:47.208930 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fz7xr_23a31617-3eb9-499e-a192-f4d24a34e835/insights-operator/1.log" Nov 01 16:54:47.237440 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:47.237401 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6x8kz_b220d197-999f-4b20-af10-f9ffb9420614/kube-rbac-proxy/0.log" Nov 01 16:54:47.261200 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:47.261164 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6x8kz_b220d197-999f-4b20-af10-f9ffb9420614/exporter/0.log" Nov 01 16:54:47.286442 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:47.286405 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6x8kz_b220d197-999f-4b20-af10-f9ffb9420614/extractor/0.log" Nov 01 16:54:47.481452 ip-10-0-143-101 kubenswrapper[4911]: I1101 16:54:47.481370 4911 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-dmh7c-gpfqf_5b478dcb-23eb-4747-8a96-05d1575acbb5/insights-gathering/0.log"